Nov 01 00:22:47.082945 ip-10-0-137-150 systemd[1]: Starting Kubernetes Kubelet... Nov 01 00:22:47.484035 ip-10-0-137-150 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:22:47.484035 ip-10-0-137-150 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 00:22:47.484035 ip-10-0-137-150 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:22:47.484035 ip-10-0-137-150 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:22:47.490130 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490040 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.490130 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490126 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.490130 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490130 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.490130 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490133 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.490130 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490137 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.490130 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490139 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490142 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490145 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490147 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490150 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490152 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490155 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490158 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490160 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490163 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490165 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490168 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490170 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490173 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490176 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490179 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490182 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490184 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490186 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490189 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.490352 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490191 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490201 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490204 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490207 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490210 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490213 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490215 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490219 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490221 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490224 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490227 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490229 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490232 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490235 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490237 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490240 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490243 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490245 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490248 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490250 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.490841 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490253 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490255 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490258 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490260 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490263 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490265 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490268 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490270 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490273 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490275 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490278 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490280 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490284 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490286 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490289 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490291 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490294 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490296 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490299 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490302 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.491305 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490306 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490311 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490315 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490318 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490321 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490324 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490327 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490330 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490332 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490335 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490337 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490340 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490342 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490345 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490347 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490350 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490352 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490355 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490358 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.491793 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490361 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490364 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490366 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490369 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490372 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490374 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490376 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490379 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490381 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490385 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490389 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490392 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490395 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490398 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490400 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490403 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490406 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490408 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490411 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.492239 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490414 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490418 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.490421 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491266 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491274 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491278 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491281 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491284 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491287 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491290 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491292 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491295 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491298 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491301 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491304 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491306 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491309 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491312 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491314 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.492700 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491318 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491322 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491325 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491328 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491331 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491335 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491338 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491341 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491343 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491346 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491349 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491351 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491354 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491357 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491360 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491362 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491365 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491367 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491370 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491372 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.493164 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491375 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491378 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491380 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491382 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491385 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491388 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491390 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491393 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491396 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491398 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491401 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491404 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491408 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491411 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491413 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491416 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491419 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491421 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491424 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.493660 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491427 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491430 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491432 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491434 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491437 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491439 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491442 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491444 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491446 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491449 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491452 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491455 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491457 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491459 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491462 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491464 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491467 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491469 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491472 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491474 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.494121 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491477 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491480 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491482 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491485 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491487 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491490 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491492 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491495 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491497 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491501 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491503 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491506 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491509 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491512 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491514 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491516 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491519 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491521 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491524 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491526 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.494591 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491529 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491531 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491534 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491536 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491539 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491541 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491543 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491546 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491548 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491550 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.491553 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492066 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492072 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492076 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492079 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492082 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492084 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492087 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492089 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.495082 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492092 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492094 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492097 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492100 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492106 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492110 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492114 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492117 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492120 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492122 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492125 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492127 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492130 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492133 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492135 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492138 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492140 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492143 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492145 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.495526 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492148 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492150 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492153 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492155 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492158 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492160 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492162 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492165 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492168 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492171 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492173 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492176 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492178 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492181 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492183 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492186 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492189 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492193 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492195 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.496003 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492198 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492200 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492203 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492205 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492207 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492210 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492212 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492215 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492217 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492220 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492222 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492224 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492227 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492229 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492232 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492234 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492236 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492239 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492241 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492244 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.496456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492246 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492249 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492253 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492255 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492258 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492260 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492262 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492265 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492267 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492270 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492274 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492276 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492279 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492281 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492283 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492286 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492289 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492291 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492294 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492296 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.496933 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492299 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492301 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492303 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492306 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492308 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492311 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492313 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492315 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492318 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492320 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492323 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492326 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492328 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492330 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492333 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492336 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492339 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492341 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492344 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.492349 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.497430 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492437 2570 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492444 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492452 2570 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492457 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492462 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492466 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492470 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492474 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492477 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492480 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492483 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492487 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492490 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492493 2570 flags.go:64] FLAG: --cgroup-root="" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492497 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492500 2570 flags.go:64] FLAG: --client-ca-file="" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492503 2570 flags.go:64] FLAG: --cloud-provider="external" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492506 2570 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492510 2570 flags.go:64] FLAG: --cluster-domain="" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492513 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492516 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492519 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492522 2570 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492526 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 00:22:47.497948 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492529 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492532 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492535 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492538 2570 flags.go:64] FLAG: --contention-profiling="false" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492542 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492545 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492549 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492551 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492556 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492559 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492564 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492566 2570 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492569 2570 flags.go:64] FLAG: --enable-server="true" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492572 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492577 2570 flags.go:64] FLAG: --event-burst="100" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492580 2570 flags.go:64] FLAG: --event-qps="50" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492583 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492586 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492589 2570 flags.go:64] FLAG: --eviction-hard="" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492594 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492596 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492599 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492604 2570 flags.go:64] FLAG: --eviction-soft="" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492607 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492610 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 00:22:47.498573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492613 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492616 2570 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492632 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492636 2570 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492638 2570 flags.go:64] FLAG: --feature-gates="" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492647 2570 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492650 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492653 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492656 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492659 2570 flags.go:64] FLAG: --healthz-port="10248" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492662 2570 flags.go:64] FLAG: --help="false" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492665 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492668 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492672 2570 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492675 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492678 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492681 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492685 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492688 2570 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492690 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492693 2570 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492696 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492699 2570 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492702 2570 flags.go:64] FLAG: --kube-reserved="" Nov 01 00:22:47.499238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492705 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492707 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492711 2570 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492714 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492716 2570 flags.go:64] FLAG: --lock-file="" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492720 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492723 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492727 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492732 2570 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492735 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492738 2570 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492741 2570 flags.go:64] FLAG: --logging-format="text" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492744 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492747 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492750 2570 flags.go:64] FLAG: --manifest-url="" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492753 2570 flags.go:64] FLAG: --manifest-url-header="" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492758 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492761 2570 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492764 2570 flags.go:64] FLAG: --max-pods="110" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492767 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492771 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492774 2570 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492777 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492780 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492783 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 00:22:47.499826 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492786 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492793 2570 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492796 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492799 2570 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492802 2570 flags.go:64] FLAG: --pod-cidr="" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492805 2570 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492808 2570 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492811 2570 flags.go:64] FLAG: --pods-per-core="0" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492814 2570 flags.go:64] FLAG: --port="10250" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492817 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492820 2570 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-076faa3b1608e5546" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492823 2570 flags.go:64] FLAG: --qos-reserved="" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492827 2570 flags.go:64] FLAG: --read-only-port="10255" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492830 2570 flags.go:64] FLAG: --register-node="true" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492833 2570 flags.go:64] FLAG: --register-with-taints="" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492837 2570 flags.go:64] FLAG: --registry-burst="10" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492840 2570 flags.go:64] FLAG: --registry-qps="5" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492842 2570 flags.go:64] FLAG: --reserved-cpus="" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492845 2570 flags.go:64] FLAG: --reserved-memory="" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492849 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492852 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492855 2570 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492858 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492861 2570 flags.go:64] FLAG: --runonce="false" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492863 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492866 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492869 2570 flags.go:64] FLAG: --seccomp-default="false" Nov 01 00:22:47.500421 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492872 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492876 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492880 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492883 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492886 2570 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492890 2570 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492893 2570 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492896 2570 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492898 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492901 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492904 2570 flags.go:64] FLAG: --system-cgroups="" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492907 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492912 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492915 2570 flags.go:64] FLAG: --tls-cert-file="" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492918 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492922 2570 flags.go:64] FLAG: --tls-min-version="" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492925 2570 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492928 2570 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492931 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492934 2570 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492937 2570 flags.go:64] FLAG: --v="2" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492941 2570 flags.go:64] FLAG: --version="false" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492945 2570 flags.go:64] FLAG: --vmodule="" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492949 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.492952 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 00:22:47.501071 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493087 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493091 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493094 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493097 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493100 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493102 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493105 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493108 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493110 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493113 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493121 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493124 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493128 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493131 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493133 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493136 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493138 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493141 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493143 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493145 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.501710 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493148 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493150 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493153 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493155 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493160 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493163 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493166 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493168 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493171 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493174 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493177 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493179 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493182 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493184 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493187 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493189 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493192 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493194 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493197 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493199 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.502231 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493202 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493204 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493207 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493210 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493214 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493218 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493220 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493223 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493226 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493229 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493231 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493234 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493237 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493239 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493243 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493247 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493251 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493254 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.502720 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493257 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493259 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493262 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493264 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493267 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493269 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493272 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493274 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493277 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493279 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493282 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493284 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493287 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493289 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493292 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493294 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493297 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493299 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493302 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493305 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.503155 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493308 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493310 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493313 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493315 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493317 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493320 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493322 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493325 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493327 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493330 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493334 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493336 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493339 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493341 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493344 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493346 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493349 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493351 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493354 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493356 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.503640 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493359 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493361 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493364 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493366 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493369 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493371 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493374 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493376 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.493401 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493435 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493439 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493443 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493445 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493448 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493450 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493453 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.504110 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493456 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493458 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493461 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493463 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493465 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493468 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493470 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493475 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493477 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493480 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493482 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493485 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493487 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493490 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493493 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493495 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493498 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493500 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493504 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.504485 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493508 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493510 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493513 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493516 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493518 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493521 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493523 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493526 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493529 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493533 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493536 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493539 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493542 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493545 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493547 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493550 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493552 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493555 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493557 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.504943 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493560 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493564 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493567 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493569 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493572 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493574 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493577 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493580 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493582 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493585 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493587 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493590 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493593 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493596 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493598 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493601 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493603 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493606 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493608 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493610 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.505382 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493614 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493617 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493633 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493640 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493643 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493646 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493648 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493651 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493653 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493655 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493658 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493661 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493663 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493669 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493671 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493674 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493676 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493679 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493682 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493684 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.505899 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493687 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493689 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493692 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493695 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493697 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493699 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493702 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493705 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493708 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493711 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493713 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493716 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493720 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493722 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493725 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493727 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493730 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493733 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493736 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493738 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.506370 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.493741 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.501908 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.501921 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.501935 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.501939 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502014 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502021 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502025 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502028 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502032 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502035 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502037 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502041 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502045 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502048 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502051 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502053 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502056 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502059 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502061 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.506852 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502064 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502066 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502069 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502072 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502075 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502078 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502081 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502083 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502086 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502089 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502092 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502095 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502097 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502100 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502103 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502106 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502108 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502111 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502113 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.507304 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502116 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502119 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502121 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502124 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502127 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502129 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502132 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502134 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502137 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502139 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502142 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502144 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502147 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502149 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502152 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502154 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502157 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502159 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502162 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.507800 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502165 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502167 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502169 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502172 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502175 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502178 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502181 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502183 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502186 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502189 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502191 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502194 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502197 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502199 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502202 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502204 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502207 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502209 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502212 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502214 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.508229 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502217 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502219 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502221 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502224 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502226 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502229 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502231 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502235 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502239 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502242 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502245 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502248 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502251 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502253 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502256 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502259 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502266 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502269 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502272 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.508704 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502275 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502278 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502281 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502283 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502286 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502288 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502291 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502293 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502296 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502298 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502300 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502303 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502305 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502308 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.502335 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:22:47.509166 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502531 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502535 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502538 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502540 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502543 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502546 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502549 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502551 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502554 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502557 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502560 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502564 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502567 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502569 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502572 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502581 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502585 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502588 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502591 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.509532 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502594 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502596 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502599 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502603 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502607 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502610 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502614 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502617 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502634 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502637 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502640 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502642 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502645 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502647 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502650 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502652 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502655 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502657 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.510043 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502660 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502663 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502665 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502668 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502670 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502673 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502675 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502678 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502681 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502684 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502686 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502694 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502697 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502700 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502703 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502705 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502708 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502711 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502713 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502715 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.510482 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502718 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502721 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502723 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502726 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502729 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502731 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502734 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502737 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502739 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502742 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502744 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502747 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502749 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502752 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502754 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502756 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502759 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502761 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502764 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502766 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.511057 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502768 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502771 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502773 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502776 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502784 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502787 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502789 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502792 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502795 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502797 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502800 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502802 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502805 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502807 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502809 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502812 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502814 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502817 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502819 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502821 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.511600 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502824 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502826 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502828 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502831 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502833 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502836 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502838 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502841 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502843 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.502866 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502915 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502920 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502924 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502927 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502930 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502933 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:22:47.512422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502935 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502943 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502946 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502949 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502951 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502954 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502956 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502959 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502961 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502963 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502966 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502968 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502971 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502973 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502976 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502978 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502981 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502983 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502985 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502988 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:22:47.513143 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502990 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.502992 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503002 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503004 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503007 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503009 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503012 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503015 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503019 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503021 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503024 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503027 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503029 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503032 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503042 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503045 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503048 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503050 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503053 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:22:47.513964 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503057 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503061 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503064 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503067 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503070 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503073 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503075 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503078 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503080 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503082 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503085 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503087 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503090 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503092 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503095 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503097 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503100 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503102 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503104 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:22:47.514972 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503107 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503110 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503112 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503115 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503118 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503120 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503123 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503125 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503127 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503135 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503138 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503141 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503143 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503145 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503148 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503150 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503152 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503155 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503157 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:22:47.515422 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503160 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503162 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503165 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503167 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503170 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503172 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503175 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503178 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503180 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503183 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503185 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503187 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503190 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503192 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503195 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503197 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503199 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503202 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503204 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503207 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:22:47.515934 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503209 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503216 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:47.503218 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.503273 2570 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.505227 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.506275 2570 server.go:1014] "Starting client certificate rotation" Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.506372 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.506407 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.509000 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:22:47.516393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.514530 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:22:47.525251 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.525238 2570 log.go:25] "Validated CRI v1 runtime API" Nov 01 00:22:47.529260 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.529245 2570 log.go:25] "Validated CRI v1 image API" Nov 01 00:22:47.530077 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.530055 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 00:22:47.536111 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.536088 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 814c003a-fcc6-40dc-abcc-4dffac607da7:/dev/nvme0n1p4 85d8d246-5371-45b5-8fc8-aabfb348aa5f:/dev/nvme0n1p3] Nov 01 00:22:47.536195 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.536109 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 00:22:47.541548 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.541445 2570 manager.go:217] Machine: {Timestamp:2026-11-01 00:22:47.539805492 +0000 UTC m=+0.344295938 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100494 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22d0086a4d0937f0c3429601604d8e SystemUUID:ec22d008-6a4d-0937-f0c3-429601604d8e BootID:f3398203-3447-4e69-b843-a1624bbe2a8e Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:fa:0c:3f:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:fa:0c:3f:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:11:63:1a:05:a1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 00:22:47.541990 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.541980 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 00:22:47.542110 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.542098 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 00:22:47.544713 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.544688 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 00:22:47.544850 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.544716 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-150.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 00:22:47.544891 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.544853 2570 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 00:22:47.544891 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.544861 2570 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 00:22:47.544950 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.544894 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:22:47.544950 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.544908 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:22:47.545614 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.545604 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 00:22:47.549326 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.549314 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 00:22:47.549452 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.549442 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 00:22:47.550216 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.550206 2570 kubelet.go:493] "Attempting to sync node with API server" Nov 01 00:22:47.550269 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.550220 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 00:22:47.550269 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.550234 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 00:22:47.550269 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.550243 2570 kubelet.go:405] "Adding apiserver pod source" Nov 01 00:22:47.550269 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.550252 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 00:22:47.550678 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.550650 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:22:47.553029 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.553011 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 00:22:47.556597 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.556585 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:22:47.557485 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.557467 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 00:22:47.557521 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.557504 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 00:22:47.558056 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558046 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 00:22:47.558093 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558062 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 00:22:47.558093 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558069 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 00:22:47.558093 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558074 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 00:22:47.558093 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558080 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 00:22:47.558093 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558087 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 00:22:47.558956 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558947 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 00:22:47.558985 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558957 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 00:22:47.558985 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558964 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 00:22:47.558985 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558970 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 00:22:47.559058 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558988 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 00:22:47.559058 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.558997 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 00:22:47.559720 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.559711 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 00:22:47.559754 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.559720 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 00:22:47.563598 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.563585 2570 server.go:1263] "Started kubelet" Nov 01 00:22:47.563688 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.563646 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 00:22:47.563762 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.563702 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 00:22:47.563816 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.563792 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 00:22:47.564376 ip-10-0-137-150 systemd[1]: Started Kubernetes Kubelet. Nov 01 00:22:47.564988 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.564857 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 00:22:47.570427 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.570407 2570 server.go:317] "Adding debug handlers to kubelet server" Nov 01 00:22:47.572894 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.572857 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 00:22:47.573852 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.573838 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 00:22:47.573916 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.573856 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 00:22:47.574257 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.574232 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:47.574580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.574560 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 00:22:47.574580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.574579 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 00:22:47.574746 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.574560 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 00:22:47.574783 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.574754 2570 reconstruct.go:97] "Volume reconstruction finished" Nov 01 00:22:47.574783 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.574764 2570 reconciler.go:29] "Reconciler: start to sync state" Nov 01 00:22:47.575779 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.575761 2570 factory.go:153] Registering CRI-O factory Nov 01 00:22:47.575871 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.575782 2570 factory.go:223] Registration of the crio container factory successfully Nov 01 00:22:47.575871 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.575831 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 00:22:47.575871 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.575840 2570 factory.go:55] Registering systemd factory Nov 01 00:22:47.575871 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.575847 2570 factory.go:223] Registration of the systemd container factory successfully Nov 01 00:22:47.575871 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.575863 2570 factory.go:103] Registering Raw factory Nov 01 00:22:47.576061 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.575878 2570 manager.go:1199] Started watching for new ooms in manager Nov 01 00:22:47.576987 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.576974 2570 manager.go:319] Starting recovery of all containers Nov 01 00:22:47.586708 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.586689 2570 manager.go:324] Recovery completed Nov 01 00:22:47.590857 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.590842 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:47.592818 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.592803 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:47.592885 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.592840 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:47.592885 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.592851 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:47.593394 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.593379 2570 cpu_manager.go:226] "Starting" policy="none" Nov 01 00:22:47.593453 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.593394 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 00:22:47.593453 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.593415 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 00:22:47.593453 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.593415 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:47.594185 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.594169 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:47.594230 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.594200 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:47.594230 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.594224 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:47.595563 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.595552 2570 policy_none.go:50] "Start" Nov 01 00:22:47.595605 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.595568 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 00:22:47.595605 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.595587 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 00:22:47.597610 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.597600 2570 policy_none.go:44] "Start" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.639214 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.641756 2570 manager.go:341] "Starting Device Plugin manager" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.641781 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.641790 2570 server.go:87] "Starting device plugin registration server" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.642323 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.642345 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.642833 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.642939 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.642948 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.643438 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.643494 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.652503 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.653835 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.653855 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.653872 2570 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 00:22:47.656700 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.653901 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 00:22:47.715864 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.715805 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-150.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 00:22:47.715864 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.715825 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-150.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 00:22:47.716847 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.715828 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-150.us-west-1.compute.internal.18e3c3fce4eb6823 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-150.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-150.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 00:22:47.563397155 +0000 UTC m=+0.367887601,LastTimestamp:2026-11-01 00:22:47.563397155 +0000 UTC m=+0.367887601,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-150.us-west-1.compute.internal,}" Nov 01 00:22:47.743375 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.743315 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:47.744878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.744857 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:47.744985 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.744893 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:47.744985 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.744909 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:47.744985 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.744940 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.755210 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.755189 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal"] Nov 01 00:22:47.755293 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.755246 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:47.756078 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.756063 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:47.756162 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.756092 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:47.756162 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.756103 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:47.758257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.758245 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:47.758406 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.758389 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.758464 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.758427 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:47.759015 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.758999 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:47.759082 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.759023 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:47.759082 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.759032 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:47.759082 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.759072 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:47.759273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.759096 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:47.759273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.759111 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:47.760955 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.760942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.760993 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.760966 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:47.761888 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.761873 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:47.761982 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.761897 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:47.761982 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.761909 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:47.776748 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.776732 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.776856 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.776754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.776856 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.776770 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0c4975681f29214750bf74479eb4135-config\") pod \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"f0c4975681f29214750bf74479eb4135\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.793059 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.793040 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.796136 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.796119 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.798006 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.797989 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 00:22:47.798068 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.798038 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mj9ff" Nov 01 00:22:47.816244 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.816218 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 00:22:47.816350 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.816332 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-150.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 00:22:47.816427 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.816388 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 00:22:47.816427 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.816357 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-150.us-west-1.compute.internal.18e3c3fce6ac728d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-150.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-150.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-150.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 00:22:47.592825485 +0000 UTC m=+0.397315931,LastTimestamp:2026-11-01 00:22:47.592825485 +0000 UTC m=+0.397315931,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-150.us-west-1.compute.internal,}" Nov 01 00:22:47.816512 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.816431 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 01 00:22:47.876873 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.876844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.876873 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.876872 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.877050 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.876889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0c4975681f29214750bf74479eb4135-config\") pod \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"f0c4975681f29214750bf74479eb4135\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.877050 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.876944 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0c4975681f29214750bf74479eb4135-config\") pod \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"f0c4975681f29214750bf74479eb4135\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.877050 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.876944 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.877050 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:47.876963 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:47.877394 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:47.877379 2570 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-150.us-west-1.compute.internal\" not found" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:48.021335 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.021256 2570 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-150.us-west-1.compute.internal\" not found" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:48.077867 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.077846 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:48.078836 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.078814 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:48.078944 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.078847 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:48.078944 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.078859 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:48.078944 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.078884 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:48.096096 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.096077 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:48.099244 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.099227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:48.137320 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.137298 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:48.137320 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.137320 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-150.us-west-1.compute.internal\": node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.253086 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.253057 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.353603 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.353535 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.453904 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.453878 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.505767 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.505745 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 00:22:48.506185 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.505884 2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://abbe3b91050ab48e2ac008ae67e6ea89-9261093490b13aca.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-150.us-west-1.compute.internal.18e3c3fce6acbdf9\": read tcp 10.0.137.150:34330->18.223.65.189:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-150.us-west-1.compute.internal.18e3c3fce6acbdf9 default 4777 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-150.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-150.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-150.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 00:22:47 +0000 UTC,LastTimestamp:2026-11-01 00:22:47.756097031 +0000 UTC m=+0.560587476,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-150.us-west-1.compute.internal,}" Nov 01 00:22:48.554211 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.554188 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.654967 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.654897 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.755892 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.755853 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.783387 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.783371 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:22:48.784282 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.784259 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:22:48.784364 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.784299 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:22:48.784364 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.784313 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:22:48.801492 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.801459 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 00:17:48 +0000 UTC" deadline="2028-07-28 14:20:47.967460858 +0000 UTC" Nov 01 00:22:48.801492 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.801486 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15253h57m59.16597759s" Nov 01 00:22:48.831715 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:48.831691 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:22:48.856847 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.856825 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:48.957154 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:48.957083 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:49.050866 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.050842 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:22:49.057902 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.057878 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:22:49.134950 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.134924 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:22:49.174192 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.174166 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:49.204913 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:49.204884 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c4975681f29214750bf74479eb4135.slice/crio-a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185 WatchSource:0}: Error finding container a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185: Status 404 returned error can't find the container with id a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185 Nov 01 00:22:49.205369 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:49.205344 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3f780546aa667fa9a0746baec62e3e.slice/crio-0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760 WatchSource:0}: Error finding container 0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760: Status 404 returned error can't find the container with id 0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760 Nov 01 00:22:49.210759 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.210744 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:22:49.237234 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.237216 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:22:49.238428 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.238415 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:22:49.303852 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.303824 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:22:49.421894 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.421864 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:22:49.551294 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.551191 2570 apiserver.go:52] "Watching apiserver" Nov 01 00:22:49.574504 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.574475 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 00:22:49.659331 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.659266 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"f0c4975681f29214750bf74479eb4135","Type":"ContainerStarted","Data":"a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185"} Nov 01 00:22:49.660880 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.660853 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"1c3f780546aa667fa9a0746baec62e3e","Type":"ContainerStarted","Data":"0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760"} Nov 01 00:22:49.662788 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.662761 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 00:22:49.663109 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.663091 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vtb29","kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal","openshift-image-registry/node-ca-kl4cn","openshift-multus/network-metrics-daemon-dtpjl","openshift-multus/multus-additional-cni-plugins-6qv8g","openshift-network-diagnostics/network-check-target-bq86m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal","openshift-multus/multus-f66hm","openshift-network-operator/iptables-alerter-gwksj","kube-system/global-pull-secret-syncer-m45bc","openshift-cluster-node-tuning-operator/tuned-g77tg","openshift-dns/node-resolver-v47fq","openshift-ovn-kubernetes/ovnkube-node-wg9vt"] Nov 01 00:22:49.667183 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.667165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.669498 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.669463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.671717 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.671697 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.674209 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.674050 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:49.674209 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.674129 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:22:49.676424 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.676400 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.678659 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.678641 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.680925 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.680908 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.683271 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.683251 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.685729 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.685689 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.686391 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686368 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.686461 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686410 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.686461 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:49.686552 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.686552 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686536 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae8165f7-bad3-418a-be14-97acb4604cf7-iptables-alerter-script\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.686666 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-serviceca\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.686750 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686690 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6l2\" (UniqueName: \"kubernetes.io/projected/38761a0d-2c71-4f3b-8d5d-39cdf321763f-kube-api-access-2j6l2\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:49.686750 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686737 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.686849 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nngm4\" (UniqueName: \"kubernetes.io/projected/2508f7ff-f9da-48fe-8e1c-ba02568e0899-kube-api-access-nngm4\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.686849 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae8165f7-bad3-418a-be14-97acb4604cf7-host-slash\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.686849 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686842 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7x8\" (UniqueName: \"kubernetes.io/projected/ae8165f7-bad3-418a-be14-97acb4604cf7-kube-api-access-rc7x8\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.686957 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686886 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-konnectivity-ca\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.686957 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-host\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.686957 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-system-cni-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.686957 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686954 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cnibin\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.687078 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.686976 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-agent-certs\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.687078 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.687014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knxln\" (UniqueName: \"kubernetes.io/projected/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-kube-api-access-knxln\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.688186 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.688168 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.688284 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.688252 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:22:49.690568 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.690538 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:49.690660 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.690600 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:22:49.693428 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.693413 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.723970 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.723940 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:22:49.724084 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.723993 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 00:22:49.724084 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.724031 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 00:22:49.724196 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.724177 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lrql\"" Nov 01 00:22:49.725895 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.725877 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 00:22:49.726039 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.726022 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 00:22:49.726085 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.726038 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m6gx8\"" Nov 01 00:22:49.727859 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.727842 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 00:22:49.728019 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.728004 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 00:22:49.728019 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.728010 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cx8lz\"" Nov 01 00:22:49.728123 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.728045 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 00:22:49.732986 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.732966 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 00:22:49.732986 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.732971 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 00:22:49.733121 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.732992 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 00:22:49.733121 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.732977 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 00:22:49.733121 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.733012 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c54w\"" Nov 01 00:22:49.733208 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.733192 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 00:22:49.734968 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.734952 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:22:49.735066 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.735053 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 00:22:49.735130 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.735119 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k669s\"" Nov 01 00:22:49.737192 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.737177 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 00:22:49.737192 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.737179 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rhkgk\"" Nov 01 00:22:49.739560 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.739543 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 00:22:49.739646 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.739564 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 00:22:49.739803 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.739788 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhlh7\"" Nov 01 00:22:49.742014 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.741998 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 00:22:49.742115 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.742041 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 00:22:49.742243 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.742225 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 00:22:49.742288 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.742253 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctdk\"" Nov 01 00:22:49.753027 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.753005 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 00:22:49.753367 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.753353 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 00:22:49.773252 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.773223 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 00:22:49.773252 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.773241 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 00:22:49.773252 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.773247 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 00:22:49.773252 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.773230 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 00:22:49.773252 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.773249 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nz8nc\"" Nov 01 00:22:49.776376 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.776355 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 00:22:49.778431 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.778415 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 00:22:49.778497 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.778469 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r4xb2" Nov 01 00:22:49.788109 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788081 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-bin\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788183 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.788183 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-kubernetes\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.788263 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788243 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tgh\" (UniqueName: \"kubernetes.io/projected/ce5465ef-bfd2-4075-8560-a19c26720d7e-kube-api-access-d6tgh\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.788304 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788276 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5fs\" (UniqueName: \"kubernetes.io/projected/9d259950-9b3c-41dc-87bd-74286a0acda0-kube-api-access-fh5fs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.788304 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788298 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-kubelet\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788395 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788324 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.788395 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.788460 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-systemd-units\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788460 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788522 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-config\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788522 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788479 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae8165f7-bad3-418a-be14-97acb4604cf7-host-slash\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.788522 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.788614 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788549 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae8165f7-bad3-418a-be14-97acb4604cf7-host-slash\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.788614 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-system-cni-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.788794 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788794 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-kubelet-config\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.788794 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788673 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-system-cni-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.788794 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788794 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788764 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-script-lib\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.788794 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788788 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.789052 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788797 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-tmp\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.789052 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.789052 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-cnibin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.789052 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-etc-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.789052 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.788965 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nngm4\" (UniqueName: \"kubernetes.io/projected/2508f7ff-f9da-48fe-8e1c-ba02568e0899-kube-api-access-nngm4\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.789052 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789002 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysconfig\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-device-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789117 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-cni-binary-copy\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-sys\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5465ef-bfd2-4075-8560-a19c26720d7e-tmp-dir\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-sys-fs\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-systemd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789239 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-netd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.789273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-netns\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-env-overrides\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhr6\" (UniqueName: \"kubernetes.io/projected/c767a0e3-268a-423f-8113-f98ab65fa4b3-kube-api-access-4hhr6\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789361 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-systemd\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-var-lib-kubelet\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-multus\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-host\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-tuned\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789462 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-host\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc2b6\" (UniqueName: \"kubernetes.io/projected/48f9765c-72a2-4a67-b161-9de71454b8cc-kube-api-access-pc2b6\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789530 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-socket-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789550 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpc6\" (UniqueName: \"kubernetes.io/projected/c79db355-84dd-4f0a-b102-d7c28b1caee6-kube-api-access-vjpc6\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-kubelet\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.789609 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789604 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-etc-kubernetes\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789621 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cnibin\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789680 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce5465ef-bfd2-4075-8560-a19c26720d7e-hosts-file\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789696 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-registration-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789713 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-bin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789729 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-netns\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789734 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cnibin\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-ovn\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-node-log\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789790 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-log-socket\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-socket-dir-parent\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.789916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.789954 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:49.790133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790010 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-conf-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.790060 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:22:50.290035703 +0000 UTC m=+3.094526167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae8165f7-bad3-418a-be14-97acb4604cf7-iptables-alerter-script\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-serviceca\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6l2\" (UniqueName: \"kubernetes.io/projected/38761a0d-2c71-4f3b-8d5d-39cdf321763f-kube-api-access-2j6l2\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790267 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-modprobe-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790407 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-run\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-k8s-cni-cncf-io\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-multus-certs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790551 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-serviceca\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7x8\" (UniqueName: \"kubernetes.io/projected/ae8165f7-bad3-418a-be14-97acb4604cf7-kube-api-access-rc7x8\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790635 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae8165f7-bad3-418a-be14-97acb4604cf7-iptables-alerter-script\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.790689 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-konnectivity-ca\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790792 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-conf\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-daemon-config\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-dbus\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-slash\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790948 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-agent-certs\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.790977 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knxln\" (UniqueName: \"kubernetes.io/projected/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-kube-api-access-knxln\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791020 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-lib-modules\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.791155 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-host\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.791564 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791187 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-system-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.791564 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791220 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-hostroot\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.791564 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791251 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-konnectivity-ca\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.791564 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791249 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 00:22:49.791564 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.791296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-var-lib-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.794674 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.794658 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-agent-certs\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.855542 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.855454 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngm4\" (UniqueName: \"kubernetes.io/projected/2508f7ff-f9da-48fe-8e1c-ba02568e0899-kube-api-access-nngm4\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:49.856356 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.856334 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knxln\" (UniqueName: \"kubernetes.io/projected/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-kube-api-access-knxln\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.856465 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.856339 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7x8\" (UniqueName: \"kubernetes.io/projected/ae8165f7-bad3-418a-be14-97acb4604cf7-kube-api-access-rc7x8\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.856533 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.856515 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6l2\" (UniqueName: \"kubernetes.io/projected/38761a0d-2c71-4f3b-8d5d-39cdf321763f-kube-api-access-2j6l2\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:49.892249 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892220 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-systemd-units\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892365 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892365 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-config\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892365 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.892365 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892365 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892339 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-systemd-units\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892365 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892349 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-kubelet-config\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892408 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892412 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-kubelet-config\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892433 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892449 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-script-lib\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892451 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892477 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-tmp\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892502 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-cnibin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-etc-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892583 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysconfig\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892626 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-cnibin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-device-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-cni-binary-copy\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-sys\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.892721 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892690 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-device-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892631 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysconfig\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892624 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-etc-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5465ef-bfd2-4075-8560-a19c26720d7e-tmp-dir\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892757 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-sys-fs\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892776 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-sys\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-systemd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.892904 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.892964 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret podName:4454de50-086b-4842-8e00-d2ffae8344d2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:50.392944961 +0000 UTC m=+3.197435400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret") pod "global-pull-secret-syncer-m45bc" (UID: "4454de50-086b-4842-8e00-d2ffae8344d2") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.892969 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-config\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-netd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893016 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-systemd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893040 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-script-lib\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893079 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-netd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-netns\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.893580 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-env-overrides\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893133 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-sys-fs\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893146 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-netns\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhr6\" (UniqueName: \"kubernetes.io/projected/c767a0e3-268a-423f-8113-f98ab65fa4b3-kube-api-access-4hhr6\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893167 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5465ef-bfd2-4075-8560-a19c26720d7e-tmp-dir\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-systemd\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893267 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893281 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-cni-binary-copy\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-var-lib-kubelet\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893317 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-systemd\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893322 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-multus\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893353 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-tuned\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893328 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-var-lib-kubelet\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893365 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-multus\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893380 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2b6\" (UniqueName: \"kubernetes.io/projected/48f9765c-72a2-4a67-b161-9de71454b8cc-kube-api-access-pc2b6\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-socket-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpc6\" (UniqueName: \"kubernetes.io/projected/c79db355-84dd-4f0a-b102-d7c28b1caee6-kube-api-access-vjpc6\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-kubelet\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.894339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-etc-kubernetes\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce5465ef-bfd2-4075-8560-a19c26720d7e-hosts-file\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-registration-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893576 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-env-overrides\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893594 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-bin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-netns\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893638 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-ovn\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893686 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-ovn\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893690 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-socket-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-node-log\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-log-socket\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893776 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-etc-kubernetes\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893774 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-registration-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893796 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-bin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893807 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-node-log\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-socket-dir-parent\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895257 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893838 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-netns\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893850 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce5465ef-bfd2-4075-8560-a19c26720d7e-hosts-file\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893859 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-socket-dir-parent\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893876 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-log-socket\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893915 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-kubelet\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-conf-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-modprobe-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.893998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-run\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-k8s-cni-cncf-io\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-multus-certs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894185 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-conf\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894218 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-run\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894233 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-multus-certs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894252 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-conf-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894362 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-modprobe-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894374 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.895853 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-daemon-config\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894495 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-conf\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-dbus\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894613 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-dbus\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894681 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-slash\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-slash\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894751 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-k8s-cni-cncf-io\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894763 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-daemon-config\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-lib-modules\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894824 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-host\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-system-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894874 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-lib-modules\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894876 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-hostroot\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894919 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-hostroot\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894940 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-host\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894920 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-var-lib-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894975 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-system-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894980 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-var-lib-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.894981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-bin\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895015 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-bin\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-kubernetes\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895091 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-kubernetes\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tgh\" (UniqueName: \"kubernetes.io/projected/ce5465ef-bfd2-4075-8560-a19c26720d7e-kube-api-access-d6tgh\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5fs\" (UniqueName: \"kubernetes.io/projected/9d259950-9b3c-41dc-87bd-74286a0acda0-kube-api-access-fh5fs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-kubelet\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895264 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-kubelet\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895443 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-tmp\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895523 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.896878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.895758 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-tuned\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.957088 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.957060 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:22:49.957172 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.957106 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:22:49.957172 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.957121 2570 projected.go:196] Error preparing data for projected volume kube-api-access-xqc5n for pod openshift-network-diagnostics/network-check-target-bq86m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:49.957252 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:49.957221 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n podName:579dfa33-9adb-4613-b055-11671039eea2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:50.457197695 +0000 UTC m=+3.261688148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xqc5n" (UniqueName: "kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n") pod "network-check-target-bq86m" (UID: "579dfa33-9adb-4613-b055-11671039eea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:49.959809 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.959776 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2b6\" (UniqueName: \"kubernetes.io/projected/48f9765c-72a2-4a67-b161-9de71454b8cc-kube-api-access-pc2b6\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:49.959902 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.959815 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpc6\" (UniqueName: \"kubernetes.io/projected/c79db355-84dd-4f0a-b102-d7c28b1caee6-kube-api-access-vjpc6\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:49.959902 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.959875 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhr6\" (UniqueName: \"kubernetes.io/projected/c767a0e3-268a-423f-8113-f98ab65fa4b3-kube-api-access-4hhr6\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:49.960570 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.960541 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tgh\" (UniqueName: \"kubernetes.io/projected/ce5465ef-bfd2-4075-8560-a19c26720d7e-kube-api-access-d6tgh\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:49.960673 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.960571 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5fs\" (UniqueName: \"kubernetes.io/projected/9d259950-9b3c-41dc-87bd-74286a0acda0-kube-api-access-fh5fs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:22:49.980897 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.980878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:22:49.987538 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.987523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:49.991976 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:49.991948 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8165f7_bad3_418a_be14_97acb4604cf7.slice/crio-05423de021baa9deb6552031a4ebf17180a97b8365774aada80623b416242df3 WatchSource:0}: Error finding container 05423de021baa9deb6552031a4ebf17180a97b8365774aada80623b416242df3: Status 404 returned error can't find the container with id 05423de021baa9deb6552031a4ebf17180a97b8365774aada80623b416242df3 Nov 01 00:22:49.993497 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:49.993482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:22:49.997182 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:49.997163 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60389df0_3bb0_42fd_9a5b_40bfa22bacf5.slice/crio-f99bf252830c0acab95b3f4507930a31d6bbf9858daa9264855613fa09492337 WatchSource:0}: Error finding container f99bf252830c0acab95b3f4507930a31d6bbf9858daa9264855613fa09492337: Status 404 returned error can't find the container with id f99bf252830c0acab95b3f4507930a31d6bbf9858daa9264855613fa09492337 Nov 01 00:22:50.003058 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:50.003036 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4ba53d_6c1a_4032_b83d_225bf88e02e8.slice/crio-5f765550379f004449a7a7c7498e53f43ee64939579ef6aeca79bd25b17f5aeb WatchSource:0}: Error finding container 5f765550379f004449a7a7c7498e53f43ee64939579ef6aeca79bd25b17f5aeb: Status 404 returned error can't find the container with id 5f765550379f004449a7a7c7498e53f43ee64939579ef6aeca79bd25b17f5aeb Nov 01 00:22:50.009372 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.009357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:22:50.011521 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.011497 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f66hm" Nov 01 00:22:50.016787 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.016770 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:22:50.018875 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.018856 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v47fq" Nov 01 00:22:50.019333 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:50.019306 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f9765c_72a2_4a67_b161_9de71454b8cc.slice/crio-c55f437815ded903957f7c3c6682c54ebd4e9e00119cc2c4f52edaa047a7dc46 WatchSource:0}: Error finding container c55f437815ded903957f7c3c6682c54ebd4e9e00119cc2c4f52edaa047a7dc46: Status 404 returned error can't find the container with id c55f437815ded903957f7c3c6682c54ebd4e9e00119cc2c4f52edaa047a7dc46 Nov 01 00:22:50.024016 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:50.023818 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d259950_9b3c_41dc_87bd_74286a0acda0.slice/crio-df5599d793d24509604a61bebdc994ce74c1c9d926a7a3da161dfed6d4ac326f WatchSource:0}: Error finding container df5599d793d24509604a61bebdc994ce74c1c9d926a7a3da161dfed6d4ac326f: Status 404 returned error can't find the container with id df5599d793d24509604a61bebdc994ce74c1c9d926a7a3da161dfed6d4ac326f Nov 01 00:22:50.026812 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.026794 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:22:50.032582 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:50.032543 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5465ef_bfd2_4075_8560_a19c26720d7e.slice/crio-80fda66eb807be3c9b964db1d67e0ad1ecbeb3ce6d6648b82c2567d4909e5d05 WatchSource:0}: Error finding container 80fda66eb807be3c9b964db1d67e0ad1ecbeb3ce6d6648b82c2567d4909e5d05: Status 404 returned error can't find the container with id 80fda66eb807be3c9b964db1d67e0ad1ecbeb3ce6d6648b82c2567d4909e5d05 Nov 01 00:22:50.039359 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:50.039335 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79db355_84dd_4f0a_b102_d7c28b1caee6.slice/crio-3a052aef8177eb60c0002f77bf693a10077a9e50abe7de0f21054ba97cd8fd68 WatchSource:0}: Error finding container 3a052aef8177eb60c0002f77bf693a10077a9e50abe7de0f21054ba97cd8fd68: Status 404 returned error can't find the container with id 3a052aef8177eb60c0002f77bf693a10077a9e50abe7de0f21054ba97cd8fd68 Nov 01 00:22:50.044939 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.044915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:22:50.054617 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:22:50.054584 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc767a0e3_268a_423f_8113_f98ab65fa4b3.slice/crio-b734bc2834027f0b0318f25842ed43adebcb55f86afe9034a00589c6ae01a414 WatchSource:0}: Error finding container b734bc2834027f0b0318f25842ed43adebcb55f86afe9034a00589c6ae01a414: Status 404 returned error can't find the container with id b734bc2834027f0b0318f25842ed43adebcb55f86afe9034a00589c6ae01a414 Nov 01 00:22:50.298179 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.298089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:50.298350 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.298289 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:50.298476 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.298445 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:22:51.298407586 +0000 UTC m=+4.102898024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:50.398906 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.398874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:50.399121 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.399002 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:50.399121 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.399051 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret podName:4454de50-086b-4842-8e00-d2ffae8344d2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:51.399038483 +0000 UTC m=+4.203528933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret") pod "global-pull-secret-syncer-m45bc" (UID: "4454de50-086b-4842-8e00-d2ffae8344d2") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:50.499846 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.499810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:50.500053 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.500021 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:22:50.500053 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.500046 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:22:50.500191 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.500059 2570 projected.go:196] Error preparing data for projected volume kube-api-access-xqc5n for pod openshift-network-diagnostics/network-check-target-bq86m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:50.500191 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:50.500128 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n podName:579dfa33-9adb-4613-b055-11671039eea2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:51.500106491 +0000 UTC m=+4.304596946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqc5n" (UniqueName: "kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n") pod "network-check-target-bq86m" (UID: "579dfa33-9adb-4613-b055-11671039eea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:50.665055 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.664955 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" event={"ID":"c79db355-84dd-4f0a-b102-d7c28b1caee6","Type":"ContainerStarted","Data":"3a052aef8177eb60c0002f77bf693a10077a9e50abe7de0f21054ba97cd8fd68"} Nov 01 00:22:50.666450 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.666416 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f66hm" event={"ID":"9d259950-9b3c-41dc-87bd-74286a0acda0","Type":"ContainerStarted","Data":"df5599d793d24509604a61bebdc994ce74c1c9d926a7a3da161dfed6d4ac326f"} Nov 01 00:22:50.667859 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.667826 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v47fq" event={"ID":"ce5465ef-bfd2-4075-8560-a19c26720d7e","Type":"ContainerStarted","Data":"80fda66eb807be3c9b964db1d67e0ad1ecbeb3ce6d6648b82c2567d4909e5d05"} Nov 01 00:22:50.669436 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.669408 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerStarted","Data":"d2375a2b5cd32d8cd8837b3ac14dc0910d62d65c1d80caaf0ef32c22accefaa7"} Nov 01 00:22:50.670991 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.670967 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g77tg" event={"ID":"48f9765c-72a2-4a67-b161-9de71454b8cc","Type":"ContainerStarted","Data":"c55f437815ded903957f7c3c6682c54ebd4e9e00119cc2c4f52edaa047a7dc46"} Nov 01 00:22:50.672884 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.672856 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kl4cn" event={"ID":"9c4ba53d-6c1a-4032-b83d-225bf88e02e8","Type":"ContainerStarted","Data":"5f765550379f004449a7a7c7498e53f43ee64939579ef6aeca79bd25b17f5aeb"} Nov 01 00:22:50.674430 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.674397 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtb29" event={"ID":"60389df0-3bb0-42fd-9a5b-40bfa22bacf5","Type":"ContainerStarted","Data":"f99bf252830c0acab95b3f4507930a31d6bbf9858daa9264855613fa09492337"} Nov 01 00:22:50.675980 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.675954 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gwksj" event={"ID":"ae8165f7-bad3-418a-be14-97acb4604cf7","Type":"ContainerStarted","Data":"05423de021baa9deb6552031a4ebf17180a97b8365774aada80623b416242df3"} Nov 01 00:22:50.677430 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.677404 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"b734bc2834027f0b0318f25842ed43adebcb55f86afe9034a00589c6ae01a414"} Nov 01 00:22:50.781787 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.781738 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:17:50 +0000 UTC" deadline="2028-05-17 15:33:08.639869348 +0000 UTC" Nov 01 00:22:50.781787 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:50.781781 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13527h10m17.858093047s" Nov 01 00:22:51.307925 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.307839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:51.308092 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.308067 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:51.308155 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.308139 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:22:53.308117867 +0000 UTC m=+6.112608304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:51.409559 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.409037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:51.409559 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.409253 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:51.409559 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.409319 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret podName:4454de50-086b-4842-8e00-d2ffae8344d2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:53.409301125 +0000 UTC m=+6.213791582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret") pod "global-pull-secret-syncer-m45bc" (UID: "4454de50-086b-4842-8e00-d2ffae8344d2") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:51.509946 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.509904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:51.510133 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.510107 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:22:51.510133 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.510127 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:22:51.510246 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.510140 2570 projected.go:196] Error preparing data for projected volume kube-api-access-xqc5n for pod openshift-network-diagnostics/network-check-target-bq86m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:51.510246 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.510204 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n podName:579dfa33-9adb-4613-b055-11671039eea2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:53.510187177 +0000 UTC m=+6.314677627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqc5n" (UniqueName: "kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n") pod "network-check-target-bq86m" (UID: "579dfa33-9adb-4613-b055-11671039eea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:51.659027 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.655972 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:51.659027 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.656153 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:22:51.659027 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.656246 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:51.659027 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.656356 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:22:51.659027 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.657876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:51.659027 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:51.658007 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:22:51.782735 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.782667 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:17:50 +0000 UTC" deadline="2028-04-18 17:15:55.117496424 +0000 UTC" Nov 01 00:22:51.782735 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:51.782734 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12832h53m3.334767416s" Nov 01 00:22:52.697909 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:52.697080 2570 generic.go:356] "Generic (PLEG): container finished" podID="1c3f780546aa667fa9a0746baec62e3e" containerID="329c71348bce591be625869ef3f40c459d9776c13a09868a7505a14d30155208" exitCode=0 Nov 01 00:22:52.697909 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:52.697138 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"1c3f780546aa667fa9a0746baec62e3e","Type":"ContainerDied","Data":"329c71348bce591be625869ef3f40c459d9776c13a09868a7505a14d30155208"} Nov 01 00:22:53.330874 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.330831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:53.331413 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.331075 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:53.331413 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.331153 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:22:57.331132711 +0000 UTC m=+10.135623148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:53.432138 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.432096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:53.432398 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.432375 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:53.432487 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.432448 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret podName:4454de50-086b-4842-8e00-d2ffae8344d2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:57.432428977 +0000 UTC m=+10.236919428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret") pod "global-pull-secret-syncer-m45bc" (UID: "4454de50-086b-4842-8e00-d2ffae8344d2") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:53.532811 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.532772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:53.532989 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.532951 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:22:53.532989 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.532980 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:22:53.533135 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.532993 2570 projected.go:196] Error preparing data for projected volume kube-api-access-xqc5n for pod openshift-network-diagnostics/network-check-target-bq86m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:53.533135 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.533112 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n podName:579dfa33-9adb-4613-b055-11671039eea2 nodeName:}" failed. No retries permitted until 2026-11-01 00:22:57.533069876 +0000 UTC m=+10.337560325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqc5n" (UniqueName: "kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n") pod "network-check-target-bq86m" (UID: "579dfa33-9adb-4613-b055-11671039eea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:53.656169 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.656035 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:53.656330 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.656171 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:22:53.656330 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.656248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:53.656330 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.656323 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:22:53.658647 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.658622 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:53.658799 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:53.658766 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:22:53.707357 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.707315 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"f0c4975681f29214750bf74479eb4135","Type":"ContainerStarted","Data":"667bc430c0b2d74090dd86db69f9aa52b152336759dcd6f7c4439e40ff10edc2"} Nov 01 00:22:53.712734 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.712559 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"1c3f780546aa667fa9a0746baec62e3e","Type":"ContainerStarted","Data":"e7ac07b1a194f408b201a44cd8faacd59c86dc6edb9d48ba2fb7d6e8cbe56bba"} Nov 01 00:22:53.832637 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.831468 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" podStartSLOduration=4.8314451720000005 podStartE2EDuration="4.831445172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:53.830904791 +0000 UTC m=+6.635395250" watchObservedRunningTime="2026-11-01 00:22:53.831445172 +0000 UTC m=+6.635935632" Nov 01 00:22:53.955494 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:53.955267 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" podStartSLOduration=4.955248199 podStartE2EDuration="4.955248199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:53.954879538 +0000 UTC m=+6.759369998" watchObservedRunningTime="2026-11-01 00:22:53.955248199 +0000 UTC m=+6.759738659" Nov 01 00:22:54.718678 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:54.718591 2570 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="c4d6c3ed301ec01dbbac0c088abc486b7b3e513a745e1387a0057444cd2852ed" exitCode=0 Nov 01 00:22:54.722273 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:54.721888 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerDied","Data":"c4d6c3ed301ec01dbbac0c088abc486b7b3e513a745e1387a0057444cd2852ed"} Nov 01 00:22:55.656288 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:55.654815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:55.656288 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:55.655080 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:22:55.656288 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:55.655165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:55.656288 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:55.655241 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:22:55.656288 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:55.655310 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:55.656288 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:55.655381 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:22:55.727469 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:55.726771 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtb29" event={"ID":"60389df0-3bb0-42fd-9a5b-40bfa22bacf5","Type":"ContainerStarted","Data":"aa658fd1be87c30ffa73f2ae9da20cd26d41fa951cf50ed1227cbb6f21bf9a93"} Nov 01 00:22:55.853476 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:55.853413 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vtb29" podStartSLOduration=2.973454691 podStartE2EDuration="7.853391939s" totalImagesPullingTime="4.879937248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:55.852279235 +0000 UTC m=+8.656769695" watchObservedRunningTime="2026-11-01 00:22:55.853391939 +0000 UTC m=+8.657882398" Nov 01 00:22:56.732025 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:56.731986 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kl4cn" event={"ID":"9c4ba53d-6c1a-4032-b83d-225bf88e02e8","Type":"ContainerStarted","Data":"fd15b892f720bfde2931e9f7e4412974c15c99a3c3a64085e3ed58033bab303f"} Nov 01 00:22:56.840317 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:56.840020 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:56.841062 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:56.841043 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:22:56.857453 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:56.856904 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kl4cn" podStartSLOduration=2.787980805 podStartE2EDuration="8.856885215s" totalImagesPullingTime="6.06890441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:56.85596749 +0000 UTC m=+9.660457951" watchObservedRunningTime="2026-11-01 00:22:56.856885215 +0000 UTC m=+9.661375675" Nov 01 00:22:57.363603 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:57.363436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:57.363874 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.363691 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:57.363874 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.363761 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:23:05.363743768 +0000 UTC m=+18.168234203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:22:57.464659 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:57.464610 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:57.464839 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.464779 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:57.464900 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.464856 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret podName:4454de50-086b-4842-8e00-d2ffae8344d2 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:05.464835947 +0000 UTC m=+18.269326400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret") pod "global-pull-secret-syncer-m45bc" (UID: "4454de50-086b-4842-8e00-d2ffae8344d2") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:22:57.565899 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:57.565862 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:57.566158 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.566137 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:22:57.566240 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.566162 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:22:57.566240 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.566174 2570 projected.go:196] Error preparing data for projected volume kube-api-access-xqc5n for pod openshift-network-diagnostics/network-check-target-bq86m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:57.566240 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.566234 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n podName:579dfa33-9adb-4613-b055-11671039eea2 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:05.566213352 +0000 UTC m=+18.370703788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqc5n" (UniqueName: "kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n") pod "network-check-target-bq86m" (UID: "579dfa33-9adb-4613-b055-11671039eea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:22:57.654644 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:57.654494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:57.654644 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.654606 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:22:57.655153 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:57.655130 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:57.655323 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.655280 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:22:57.655420 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:57.655363 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:57.655500 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:57.655482 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:22:57.739368 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:57.738905 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" event={"ID":"c79db355-84dd-4f0a-b102-d7c28b1caee6","Type":"ContainerStarted","Data":"0446828cbc9d52a700fae3838e8c346031320f4c26baf4a46e80d2151448109d"} Nov 01 00:22:58.743146 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:58.743117 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 00:22:58.744183 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:58.744153 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f66hm" event={"ID":"9d259950-9b3c-41dc-87bd-74286a0acda0","Type":"ContainerStarted","Data":"a9e0a51d1b6521be38dd469a8ac8e52cb888708fea418b8caa5000447992fb9d"} Nov 01 00:22:58.882019 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:58.881940 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 00:22:59.655018 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.654919 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T00:22:58.881960703Z","UUID":"aecc74f7-bc2e-4bc4-82ce-f934978c9c48","Handler":null,"Name":"","Endpoint":""} Nov 01 00:22:59.656698 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.656671 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 00:22:59.656698 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.656705 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 00:22:59.660100 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.660064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:22:59.660100 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.660080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:22:59.660289 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:59.660183 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:22:59.660353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.660300 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:22:59.660454 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:59.660427 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:22:59.660548 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:22:59.660531 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:22:59.750364 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.750324 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" event={"ID":"c79db355-84dd-4f0a-b102-d7c28b1caee6","Type":"ContainerStarted","Data":"a53e25b562798c86eb97c7d28159203272836acfa1b9dd774d357e89f033696f"} Nov 01 00:22:59.752804 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.752773 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v47fq" event={"ID":"ce5465ef-bfd2-4075-8560-a19c26720d7e","Type":"ContainerStarted","Data":"9424ff5c616535e8153b5aa495c9cbdd95c569d8aa7ab4adda14bf81de1398f9"} Nov 01 00:22:59.756144 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.756119 2570 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="17eabcbf35d474a9b69d9afd5af6e5165a9775e8b93078037cfa732b8eb29ddd" exitCode=0 Nov 01 00:22:59.756249 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.756221 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerDied","Data":"17eabcbf35d474a9b69d9afd5af6e5165a9775e8b93078037cfa732b8eb29ddd"} Nov 01 00:22:59.868287 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.868218 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-f66hm" podStartSLOduration=3.295681087 podStartE2EDuration="11.868199906s" totalImagesPullingTime="8.572518819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:58.861954661 +0000 UTC m=+11.666445118" watchObservedRunningTime="2026-11-01 00:22:59.868199906 +0000 UTC m=+12.672690365" Nov 01 00:22:59.868599 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:22:59.868550 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v47fq" podStartSLOduration=1.717939425 podStartE2EDuration="10.868538241s" totalImagesPullingTime="9.150598816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:22:59.867755744 +0000 UTC m=+12.672246200" watchObservedRunningTime="2026-11-01 00:22:59.868538241 +0000 UTC m=+12.673028697" Nov 01 00:23:00.760193 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:00.759953 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gwksj" event={"ID":"ae8165f7-bad3-418a-be14-97acb4604cf7","Type":"ContainerStarted","Data":"063e9cd8dd50682dcb602dbb8e7cb10896b6bd842cec73d7a20fdfeac4506097"} Nov 01 00:23:00.762930 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:00.762891 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" event={"ID":"c79db355-84dd-4f0a-b102-d7c28b1caee6","Type":"ContainerStarted","Data":"534f2bf7efb70024ac89269d5857fa4083a65c4e34ccdf4f5a5a8da444c245e9"} Nov 01 00:23:00.876665 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:00.876592 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gwksj" podStartSLOduration=3.887743762 podStartE2EDuration="12.87657411s" totalImagesPullingTime="8.988830348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:00.876178008 +0000 UTC m=+13.680668464" watchObservedRunningTime="2026-11-01 00:23:00.87657411 +0000 UTC m=+13.681064566" Nov 01 00:23:00.994923 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:00.994824 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" podStartSLOduration=2.242524086 podStartE2EDuration="11.994804566s" totalImagesPullingTime="9.75228048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:00.993934019 +0000 UTC m=+13.798424476" watchObservedRunningTime="2026-11-01 00:23:00.994804566 +0000 UTC m=+13.799295022" Nov 01 00:23:01.655065 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:01.655037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:01.655065 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:01.655053 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:01.655201 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:01.655177 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:01.655262 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:01.655247 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:01.655304 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:01.655294 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:01.655363 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:01.655349 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:01.767108 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:01.767075 2570 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="4be3322458df8ec48fd1cfd154c336dd4d26ee29448854246db24753451f1bb8" exitCode=0 Nov 01 00:23:01.767703 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:01.767167 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerDied","Data":"4be3322458df8ec48fd1cfd154c336dd4d26ee29448854246db24753451f1bb8"} Nov 01 00:23:01.768850 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:01.768811 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g77tg" event={"ID":"48f9765c-72a2-4a67-b161-9de71454b8cc","Type":"ContainerStarted","Data":"87c7756ea58b1401bfa52468cbe2359c15a4c4f700230091dcccd805509d6d7d"} Nov 01 00:23:02.091118 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:02.091056 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g77tg" podStartSLOduration=2.564550786 podStartE2EDuration="14.091037731s" totalImagesPullingTime="11.526486945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:02.090805613 +0000 UTC m=+14.895296069" watchObservedRunningTime="2026-11-01 00:23:02.091037731 +0000 UTC m=+14.895528186" Nov 01 00:23:03.654400 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:03.654370 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:03.654400 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:03.654380 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:03.654982 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:03.654500 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:03.654982 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:03.654520 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:03.654982 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:03.654587 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:03.654982 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:03.654656 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:03.775767 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:03.775731 2570 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="cf4be57ac545d98e3027f56468c5dbb4017a90f793840f013334348dec6ee9b0" exitCode=0 Nov 01 00:23:03.775941 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:03.775793 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerDied","Data":"cf4be57ac545d98e3027f56468c5dbb4017a90f793840f013334348dec6ee9b0"} Nov 01 00:23:05.427716 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:05.427470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:05.428163 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.427646 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:05.428163 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.427878 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:23:21.427856009 +0000 UTC m=+34.232346456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 00:23:05.528056 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:05.528023 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:05.528219 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.528188 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:05.528268 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.528256 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret podName:4454de50-086b-4842-8e00-d2ffae8344d2 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:21.528235302 +0000 UTC m=+34.332725742 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret") pod "global-pull-secret-syncer-m45bc" (UID: "4454de50-086b-4842-8e00-d2ffae8344d2") : object "kube-system"/"original-pull-secret" not registered Nov 01 00:23:05.628513 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:05.628474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:05.628696 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.628656 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 00:23:05.628696 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.628674 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 00:23:05.628696 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.628686 2570 projected.go:196] Error preparing data for projected volume kube-api-access-xqc5n for pod openshift-network-diagnostics/network-check-target-bq86m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:05.628815 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.628749 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n podName:579dfa33-9adb-4613-b055-11671039eea2 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:21.628731212 +0000 UTC m=+34.433221649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqc5n" (UniqueName: "kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n") pod "network-check-target-bq86m" (UID: "579dfa33-9adb-4613-b055-11671039eea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 00:23:05.654645 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:05.654599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:05.654645 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:05.654614 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:05.654850 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:05.654729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:05.654850 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.654750 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:05.654850 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.654816 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:05.654954 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:05.654907 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:05.782600 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:05.782504 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"7d3fafd43121a445b6c7d7fac2ff43a6c5532bf3b7bf2fa317c49659a48d6fa4"} Nov 01 00:23:06.787776 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:06.787585 2570 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="32d83023b7fda1bb227cf52f5a50e5b25ec8ce53a32c0469c4ea657be1fe46b2" exitCode=0 Nov 01 00:23:06.788196 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:06.787658 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerDied","Data":"32d83023b7fda1bb227cf52f5a50e5b25ec8ce53a32c0469c4ea657be1fe46b2"} Nov 01 00:23:06.790868 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:06.790846 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"1e35492af75fc71ed393e238e79befbc2375198f2ce2d916bf8d694fa5a6df2a"} Nov 01 00:23:06.790936 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:06.790909 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"8e51ddcf6103fb6bfeab6dab7a1b83f9ced542858e3cfb27172d83dd7f91bf37"} Nov 01 00:23:06.790996 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:06.790940 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"6352066690fa410f9a58a9eb4df3e65fe8aaa249cf1407bb6e830ab42ce74024"} Nov 01 00:23:06.790996 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:06.790968 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"73615d80ec12bb8bf29812b95b718dd95cde5c2aeff9058c6dbb4a35c85e09e5"} Nov 01 00:23:06.790996 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:06.790994 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"5411cdb63f024e03409064371a16be3aa807aef58c190fa1fdfa831acb697509"} Nov 01 00:23:07.654846 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:07.654817 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:07.654983 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:07.654898 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:07.655019 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:07.654998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:07.655107 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:07.655088 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:07.655164 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:07.655141 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:07.655194 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:07.655187 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:07.796445 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:07.796412 2570 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="42be5f0e621e51fb92f2c452e89fd765262d03f2142af635a9f08ae3ab094a8d" exitCode=0 Nov 01 00:23:07.796914 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:07.796471 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerDied","Data":"42be5f0e621e51fb92f2c452e89fd765262d03f2142af635a9f08ae3ab094a8d"} Nov 01 00:23:08.801937 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:08.801906 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" event={"ID":"2508f7ff-f9da-48fe-8e1c-ba02568e0899","Type":"ContainerStarted","Data":"b7b3404c87b00d047f6a5f01bfd2ed35314658ee903a896f5a4e0c9cc79a36ec"} Nov 01 00:23:08.808557 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:08.808530 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"bd2d0fbe3c09ce5623520dc43b5382eb4e9f60e932a306312d6c2618fb5a93aa"} Nov 01 00:23:08.923096 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:08.923045 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6qv8g" podStartSLOduration=7.275396808 podStartE2EDuration="20.923031831s" totalImagesPullingTime="13.647635023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:08.92226822 +0000 UTC m=+21.726758674" watchObservedRunningTime="2026-11-01 00:23:08.923031831 +0000 UTC m=+21.727522286" Nov 01 00:23:09.654831 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:09.654803 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:09.654831 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:09.654828 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:09.654995 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:09.654905 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:09.654995 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:09.654963 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:09.655050 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:09.655038 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:09.655144 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:09.655120 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:10.816040 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:10.815836 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" event={"ID":"c767a0e3-268a-423f-8113-f98ab65fa4b3","Type":"ContainerStarted","Data":"e013280b8f1961df7f726d7f8a81476554f53883653f1841955f02c5306abbb5"} Nov 01 00:23:10.816765 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:10.816072 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:10.816765 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:10.816106 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:10.832143 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:10.832113 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:11.092555 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:11.092428 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" podStartSLOduration=6.477167971 podStartE2EDuration="22.092415024s" totalImagesPullingTime="15.615247053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 00:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:11.092159407 +0000 UTC m=+23.896649863" watchObservedRunningTime="2026-11-01 00:23:11.092415024 +0000 UTC m=+23.896905479" Nov 01 00:23:11.654961 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:11.654923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:11.654961 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:11.654948 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:11.655161 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:11.655024 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:11.655161 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:11.655090 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:11.655242 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:11.655213 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:11.655285 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:11.655243 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:11.822463 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:11.822380 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:11.837536 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:11.837511 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:12.463032 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.462999 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:23:12.463192 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.463160 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 00:23:12.463708 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.463689 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:23:12.974597 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.974566 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m45bc"] Nov 01 00:23:12.975045 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.974717 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:12.975045 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:12.974829 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:12.978238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.978216 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bq86m"] Nov 01 00:23:12.978364 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.978295 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:12.978433 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:12.978383 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:12.980295 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.980275 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dtpjl"] Nov 01 00:23:12.980389 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:12.980368 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:12.980491 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:12.980473 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:14.655088 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:14.655052 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:14.655494 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:14.655113 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:14.655494 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:14.655138 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:14.655494 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:14.655250 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:14.655494 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:14.655285 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:14.655494 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:14.655345 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:16.654043 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:16.654014 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:16.654500 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:16.654015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:16.654500 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:16.654119 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtpjl" podUID="38761a0d-2c71-4f3b-8d5d-39cdf321763f" Nov 01 00:23:16.654500 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:16.654015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:16.654500 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:16.654203 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m45bc" podUID="4454de50-086b-4842-8e00-d2ffae8344d2" Nov 01 00:23:16.654500 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:16.654284 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq86m" podUID="579dfa33-9adb-4613-b055-11671039eea2" Nov 01 00:23:18.468654 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.468554 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeReady" Nov 01 00:23:18.469000 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.468695 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 00:23:18.587464 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.587208 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fxsjd"] Nov 01 00:23:18.590259 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.590237 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" Nov 01 00:23:18.637949 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.637908 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4"] Nov 01 00:23:18.640803 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.640776 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gsr76"] Nov 01 00:23:18.640922 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.640907 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" Nov 01 00:23:18.644944 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.644921 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds"] Nov 01 00:23:18.645054 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.645038 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gsr76" Nov 01 00:23:18.648063 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.648045 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5"] Nov 01 00:23:18.648170 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.648156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.651095 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.651078 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7btpl"] Nov 01 00:23:18.651206 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.651189 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.654005 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.653984 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78f6fdc666-mgj5x"] Nov 01 00:23:18.654094 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.654022 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:18.654094 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.654023 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:18.654176 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.654097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:18.654212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.654182 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7btpl" Nov 01 00:23:18.656991 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.656974 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb"] Nov 01 00:23:18.657098 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.657083 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.660081 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.660034 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7vncn"] Nov 01 00:23:18.660151 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.660143 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.662950 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.662930 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-srh5n"] Nov 01 00:23:18.663049 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.663017 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:18.666394 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.666375 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-649fb6d9c5-llkn8"] Nov 01 00:23:18.666500 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.666449 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.669844 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.669828 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.687175 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.686974 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.687175 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.687060 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.690899 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.690873 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nq7f9"] Nov 01 00:23:18.694544 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.694525 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-z4ffh\"" Nov 01 00:23:18.695462 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.695443 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.695562 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.695547 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz"] Nov 01 00:23:18.699928 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.699907 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69cbfc858f-rq59s"] Nov 01 00:23:18.703573 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.703289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.704393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.704371 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm"] Nov 01 00:23:18.704820 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.704803 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.707953 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.707933 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bvzrd"] Nov 01 00:23:18.708106 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.708085 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" Nov 01 00:23:18.711183 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.711166 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5hhd5"] Nov 01 00:23:18.711285 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.711266 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.713770 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713753 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fxsjd"] Nov 01 00:23:18.713845 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713782 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4"] Nov 01 00:23:18.713845 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713800 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gsr76"] Nov 01 00:23:18.713845 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713816 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7btpl"] Nov 01 00:23:18.713845 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713830 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5"] Nov 01 00:23:18.713845 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713841 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:18.714018 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713844 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds"] Nov 01 00:23:18.714018 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713970 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb"] Nov 01 00:23:18.714018 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.713993 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-srh5n"] Nov 01 00:23:18.714018 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.714012 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-649fb6d9c5-llkn8"] Nov 01 00:23:18.729458 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.729554 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729474 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thc6l\" (UniqueName: \"kubernetes.io/projected/5cfa57d0-764b-42fb-ad36-9ca6dc472766-kube-api-access-thc6l\") pod \"kas-connection-checker-685895c7c9-fxsjd\" (UID: \"5cfa57d0-764b-42fb-ad36-9ca6dc472766\") " pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" Nov 01 00:23:18.729554 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/783ad43d-a72e-4b58-b403-00eab40f6cdc-serving-cert\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.729651 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729600 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80541015-4203-4621-bba9-d737aa2f3266-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.729651 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcznv\" (UniqueName: \"kubernetes.io/projected/80541015-4203-4621-bba9-d737aa2f3266-kube-api-access-tcznv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.729768 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729685 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.729823 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.729823 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwpgz\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.729919 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729825 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/783ad43d-a72e-4b58-b403-00eab40f6cdc-config\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.729975 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80541015-4203-4621-bba9-d737aa2f3266-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.729975 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.729948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5xs\" (UniqueName: \"kubernetes.io/projected/c5b7952d-24a8-40c5-a755-3aeb8269714b-kube-api-access-hh5xs\") pod \"kas-connection-checker-685895c7c9-7btpl\" (UID: \"c5b7952d-24a8-40c5-a755-3aeb8269714b\") " pod="kube-system/kas-connection-checker-685895c7c9-7btpl" Nov 01 00:23:18.730063 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9gc8\" (UniqueName: \"kubernetes.io/projected/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc-kube-api-access-v9gc8\") pod \"volume-data-source-validator-58c9f5f46d-7x6c4\" (UID: \"70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" Nov 01 00:23:18.730099 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.730099 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.730167 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.730167 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.730237 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.730237 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730186 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzfvp\" (UniqueName: \"kubernetes.io/projected/b7e4c8da-dd72-438d-bd5b-a81728051a95-kube-api-access-dzfvp\") pod \"kas-connection-checker-685895c7c9-gsr76\" (UID: \"b7e4c8da-dd72-438d-bd5b-a81728051a95\") " pod="kube-system/kas-connection-checker-685895c7c9-gsr76" Nov 01 00:23:18.730309 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.730309 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730264 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtb9\" (UniqueName: \"kubernetes.io/projected/5dc70a14-ff5e-4088-ab14-15700a98ad22-kube-api-access-cjtb9\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.730383 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.730364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhqnt\" (UniqueName: \"kubernetes.io/projected/783ad43d-a72e-4b58-b403-00eab40f6cdc-kube-api-access-vhqnt\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.737473 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.737452 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nq7f9"] Nov 01 00:23:18.743200 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.743175 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 00:23:18.743314 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.743299 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.743946 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.743924 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.744051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744022 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.744135 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744120 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 00:23:18.744187 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744165 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.744238 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744212 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.744321 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744304 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.744400 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744372 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.744567 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744544 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.744647 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744609 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.744728 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744712 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 00:23:18.744786 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744769 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 00:23:18.744837 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744791 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.744837 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744836 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.744936 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744862 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 00:23:18.744936 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.744882 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.795383 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.795353 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 00:23:18.795798 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.795775 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 00:23:18.795971 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.795952 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 00:23:18.796267 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.796251 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.796746 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797091 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797369 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797415 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797671 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797729 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5tc5v\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797806 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797858 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 00:23:18.798051 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.797890 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 00:23:18.798515 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.798068 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kfxmw\"" Nov 01 00:23:18.798515 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.798076 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 00:23:18.798515 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.798069 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwm8b\"" Nov 01 00:23:18.799357 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.799332 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7vncn"] Nov 01 00:23:18.800495 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.800475 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hhd5"] Nov 01 00:23:18.801772 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.801757 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f6fdc666-mgj5x"] Nov 01 00:23:18.802373 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802360 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bvzrd"] Nov 01 00:23:18.802835 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802808 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802842 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802866 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2d2tf\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802878 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7ckpz\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802894 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9sbx\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802904 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2pkrq\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802926 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802928 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ntf5k\"" Nov 01 00:23:18.802933 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802938 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cvjqf\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802818 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802947 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t74k4\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802965 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gb7fs\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802972 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rjfbh\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802968 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-km6f6\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.802878 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.803004 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lfxl8\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.803025 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm"] Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.803027 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8q2z5\"" Nov 01 00:23:18.803212 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.803051 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 00:23:18.803655 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.803641 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz"] Nov 01 00:23:18.809684 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.809664 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69cbfc858f-rq59s"] Nov 01 00:23:18.830899 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.830868 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.831020 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.830903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4cab1f4-47e1-457c-a182-692b3be9bd16-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:18.831020 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.830919 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.831020 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.831173 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831039 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzfvp\" (UniqueName: \"kubernetes.io/projected/b7e4c8da-dd72-438d-bd5b-a81728051a95-kube-api-access-dzfvp\") pod \"kas-connection-checker-685895c7c9-gsr76\" (UID: \"b7e4c8da-dd72-438d-bd5b-a81728051a95\") " pod="kube-system/kas-connection-checker-685895c7c9-gsr76" Nov 01 00:23:18.831173 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.831173 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghrw\" (UniqueName: \"kubernetes.io/projected/252a2753-2639-4920-afaf-0fb0ca316d26-kube-api-access-wghrw\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.831173 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.831114 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:18.831358 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-cert\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:18.831358 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.831200 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.331185119 +0000 UTC m=+32.135675552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:18.831358 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-config\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.831358 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831255 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.831358 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.831339 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:18.831358 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.831355 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:18.831669 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.831393 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.331379979 +0000 UTC m=+32.135870432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:18.831669 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831350 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.831669 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/252a2753-2639-4920-afaf-0fb0ca316d26-serving-cert\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.831669 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzdn\" (UniqueName: \"kubernetes.io/projected/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-kube-api-access-jdzdn\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:18.831669 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.831669 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac18260-596d-49f0-b109-d5750e8c3644-serving-cert\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.831669 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcznv\" (UniqueName: \"kubernetes.io/projected/80541015-4203-4621-bba9-d737aa2f3266-kube-api-access-tcznv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831746 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d98af5-fb06-47ca-a462-8f974bbfd046-tmp-dir\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831856 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwpgz\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/783ad43d-a72e-4b58-b403-00eab40f6cdc-config\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztd6c\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.831994 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gc8\" (UniqueName: \"kubernetes.io/projected/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc-kube-api-access-v9gc8\") pod \"volume-data-source-validator-58c9f5f46d-7x6c4\" (UID: \"70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.831997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsw6b\" (UniqueName: \"kubernetes.io/projected/0d85dc68-06e8-4036-93b1-ececa1a779c7-kube-api-access-lsw6b\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5xs\" (UniqueName: \"kubernetes.io/projected/c5b7952d-24a8-40c5-a755-3aeb8269714b-kube-api-access-hh5xs\") pod \"kas-connection-checker-685895c7c9-7btpl\" (UID: \"c5b7952d-24a8-40c5-a755-3aeb8269714b\") " pod="kube-system/kas-connection-checker-685895c7c9-7btpl" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832054 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832075 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8m6\" (UniqueName: \"kubernetes.io/projected/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-kube-api-access-dh8m6\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832156 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832163 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832238 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832347 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtb9\" (UniqueName: \"kubernetes.io/projected/5dc70a14-ff5e-4088-ab14-15700a98ad22-kube-api-access-cjtb9\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.832382 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832383 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-snapshots\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832433 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-service-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92wl\" (UniqueName: \"kubernetes.io/projected/a9d98af5-fb06-47ca-a462-8f974bbfd046-kube-api-access-t92wl\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832749 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d98af5-fb06-47ca-a462-8f974bbfd046-config-volume\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832820 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-tmp\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thc6l\" (UniqueName: \"kubernetes.io/projected/5cfa57d0-764b-42fb-ad36-9ca6dc472766-kube-api-access-thc6l\") pod \"kas-connection-checker-685895c7c9-fxsjd\" (UID: \"5cfa57d0-764b-42fb-ad36-9ca6dc472766\") " pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/783ad43d-a72e-4b58-b403-00eab40f6cdc-serving-cert\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832913 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-stats-auth\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.832945 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80541015-4203-4621-bba9-d737aa2f3266-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832957 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.832983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqnt\" (UniqueName: \"kubernetes.io/projected/783ad43d-a72e-4b58-b403-00eab40f6cdc-kube-api-access-vhqnt\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/783ad43d-a72e-4b58-b403-00eab40f6cdc-config\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833217 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-default-certificate\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rnk\" (UniqueName: \"kubernetes.io/projected/8b062870-dd5d-4f35-b400-ea17ebd11878-kube-api-access-t6rnk\") pod \"network-check-source-6f78d59f99-vhjfm\" (UID: \"8b062870-dd5d-4f35-b400-ea17ebd11878\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80541015-4203-4621-bba9-d737aa2f3266-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833375 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.833489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.833892 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.833494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7vb\" (UniqueName: \"kubernetes.io/projected/0ac18260-596d-49f0-b109-d5750e8c3644-kube-api-access-8k7vb\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.834105 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.834083 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80541015-4203-4621-bba9-d737aa2f3266-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.834346 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.834320 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.835469 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.835447 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/783ad43d-a72e-4b58-b403-00eab40f6cdc-serving-cert\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.835645 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.835610 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80541015-4203-4621-bba9-d737aa2f3266-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.835761 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.835688 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.836297 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.836275 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.843044 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843009 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 00:23:18.843152 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843039 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 00:23:18.843152 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843054 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.843152 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843075 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.843152 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843100 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 00:23:18.843152 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843104 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.843152 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843131 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.843152 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843139 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.843430 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843187 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 00:23:18.843430 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843226 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.843430 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.843375 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 00:23:18.844176 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.844156 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 00:23:18.844459 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.844447 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 00:23:18.892804 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.892738 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzfvp\" (UniqueName: \"kubernetes.io/projected/b7e4c8da-dd72-438d-bd5b-a81728051a95-kube-api-access-dzfvp\") pod \"kas-connection-checker-685895c7c9-gsr76\" (UID: \"b7e4c8da-dd72-438d-bd5b-a81728051a95\") " pod="kube-system/kas-connection-checker-685895c7c9-gsr76" Nov 01 00:23:18.893956 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.893908 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcznv\" (UniqueName: \"kubernetes.io/projected/80541015-4203-4621-bba9-d737aa2f3266-kube-api-access-tcznv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.894800 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.894756 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqnt\" (UniqueName: \"kubernetes.io/projected/783ad43d-a72e-4b58-b403-00eab40f6cdc-kube-api-access-vhqnt\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:18.894909 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.894825 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtb9\" (UniqueName: \"kubernetes.io/projected/5dc70a14-ff5e-4088-ab14-15700a98ad22-kube-api-access-cjtb9\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:18.894967 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.894942 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.895070 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.895046 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5xs\" (UniqueName: \"kubernetes.io/projected/c5b7952d-24a8-40c5-a755-3aeb8269714b-kube-api-access-hh5xs\") pod \"kas-connection-checker-685895c7c9-7btpl\" (UID: \"c5b7952d-24a8-40c5-a755-3aeb8269714b\") " pod="kube-system/kas-connection-checker-685895c7c9-7btpl" Nov 01 00:23:18.895287 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.895268 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwpgz\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:18.895413 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.895396 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thc6l\" (UniqueName: \"kubernetes.io/projected/5cfa57d0-764b-42fb-ad36-9ca6dc472766-kube-api-access-thc6l\") pod \"kas-connection-checker-685895c7c9-fxsjd\" (UID: \"5cfa57d0-764b-42fb-ad36-9ca6dc472766\") " pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" Nov 01 00:23:18.896929 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.896909 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gc8\" (UniqueName: \"kubernetes.io/projected/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc-kube-api-access-v9gc8\") pod \"volume-data-source-validator-58c9f5f46d-7x6c4\" (UID: \"70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" Nov 01 00:23:18.904387 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.904371 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" Nov 01 00:23:18.934683 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934660 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.934801 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-default-certificate\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.934801 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rnk\" (UniqueName: \"kubernetes.io/projected/8b062870-dd5d-4f35-b400-ea17ebd11878-kube-api-access-t6rnk\") pod \"network-check-source-6f78d59f99-vhjfm\" (UID: \"8b062870-dd5d-4f35-b400-ea17ebd11878\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" Nov 01 00:23:18.934916 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.934916 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.935011 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7vb\" (UniqueName: \"kubernetes.io/projected/0ac18260-596d-49f0-b109-d5750e8c3644-kube-api-access-8k7vb\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.935011 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4cab1f4-47e1-457c-a182-692b3be9bd16-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:18.935011 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.934977 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:18.935160 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.935045 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.43502614 +0000 UTC m=+32.239516589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:18.935160 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.934981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.935276 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.935276 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935224 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghrw\" (UniqueName: \"kubernetes.io/projected/252a2753-2639-4920-afaf-0fb0ca316d26-kube-api-access-wghrw\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.935276 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.935135 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:18.935552 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.935315 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.435299828 +0000 UTC m=+32.239790263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:18.935552 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-cert\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:18.935552 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935364 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-config\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.935771 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935616 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.935842 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.935898 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/252a2753-2639-4920-afaf-0fb0ca316d26-serving-cert\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.935898 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzdn\" (UniqueName: \"kubernetes.io/projected/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-kube-api-access-jdzdn\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:18.935998 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac18260-596d-49f0-b109-d5750e8c3644-serving-cert\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.935998 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.935998 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.935988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:18.936139 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936002 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4cab1f4-47e1-457c-a182-692b3be9bd16-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:18.936139 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936012 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d98af5-fb06-47ca-a462-8f974bbfd046-tmp-dir\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.936139 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936064 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd6c\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.936139 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsw6b\" (UniqueName: \"kubernetes.io/projected/0d85dc68-06e8-4036-93b1-ececa1a779c7-kube-api-access-lsw6b\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.936139 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8m6\" (UniqueName: \"kubernetes.io/projected/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-kube-api-access-dh8m6\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936265 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936287 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-snapshots\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936324 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-service-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.936369 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936351 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.936775 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92wl\" (UniqueName: \"kubernetes.io/projected/a9d98af5-fb06-47ca-a462-8f974bbfd046-kube-api-access-t92wl\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.936775 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.936775 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936412 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d98af5-fb06-47ca-a462-8f974bbfd046-tmp-dir\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.936775 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.936471 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-config\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.936956 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.936920 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:18.937010 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.936967 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:18.937010 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.936976 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.436959773 +0000 UTC m=+32.241450224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:18.937010 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.936981 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:18.937010 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.936994 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:18.937194 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.937021 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.437005146 +0000 UTC m=+32.241495594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:18.937194 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.937041 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.437030524 +0000 UTC m=+32.241520960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:18.937706 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:18.937683 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:19.437669455 +0000 UTC m=+32.242159908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:18.937803 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.937729 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-snapshots\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.937989 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.937967 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:18.938261 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938237 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.938364 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938338 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.938440 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938425 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.938524 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.938578 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938546 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d98af5-fb06-47ca-a462-8f974bbfd046-config-volume\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.938663 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-tmp\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.938663 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-stats-auth\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.938764 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.938685 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-service-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.939120 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.939095 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-tmp\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.939279 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.939254 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d98af5-fb06-47ca-a462-8f974bbfd046-config-volume\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:18.939494 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.939477 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.939896 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.939874 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-cert\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:18.940053 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.940032 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-default-certificate\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.940337 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.940322 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:18.940739 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.940710 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac18260-596d-49f0-b109-d5750e8c3644-serving-cert\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:18.941493 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.941472 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/252a2753-2639-4920-afaf-0fb0ca316d26-serving-cert\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:18.941568 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.941554 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-stats-auth\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:18.953402 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.953384 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" Nov 01 00:23:18.960173 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.960152 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gsr76" Nov 01 00:23:18.978935 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.978879 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:18.996873 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:18.996852 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7btpl" Nov 01 00:23:19.000638 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.000586 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:19.001026 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.000997 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8m6\" (UniqueName: \"kubernetes.io/projected/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-kube-api-access-dh8m6\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:19.001225 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.001197 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghrw\" (UniqueName: \"kubernetes.io/projected/252a2753-2639-4920-afaf-0fb0ca316d26-kube-api-access-wghrw\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:19.002079 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.002044 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7vb\" (UniqueName: \"kubernetes.io/projected/0ac18260-596d-49f0-b109-d5750e8c3644-kube-api-access-8k7vb\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:19.002344 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.002240 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rnk\" (UniqueName: \"kubernetes.io/projected/8b062870-dd5d-4f35-b400-ea17ebd11878-kube-api-access-t6rnk\") pod \"network-check-source-6f78d59f99-vhjfm\" (UID: \"8b062870-dd5d-4f35-b400-ea17ebd11878\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" Nov 01 00:23:19.002344 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.002302 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92wl\" (UniqueName: \"kubernetes.io/projected/a9d98af5-fb06-47ca-a462-8f974bbfd046-kube-api-access-t92wl\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:19.006370 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.003245 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztd6c\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:19.006370 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.004369 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzdn\" (UniqueName: \"kubernetes.io/projected/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-kube-api-access-jdzdn\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:19.006370 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.006085 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsw6b\" (UniqueName: \"kubernetes.io/projected/0d85dc68-06e8-4036-93b1-ececa1a779c7-kube-api-access-lsw6b\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:19.024751 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.023055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:19.035335 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.034020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:19.062600 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.051289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:19.081052 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.080310 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" Nov 01 00:23:19.091038 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.088823 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:19.297798 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.297766 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fxsjd"] Nov 01 00:23:19.300971 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.300940 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfa57d0_764b_42fb_ad36_9ca6dc472766.slice/crio-fca98fa50f03b97afd456d5cc8bab5898efb5e4b3bf5bfb529b7928b124efe12 WatchSource:0}: Error finding container fca98fa50f03b97afd456d5cc8bab5898efb5e4b3bf5bfb529b7928b124efe12: Status 404 returned error can't find the container with id fca98fa50f03b97afd456d5cc8bab5898efb5e4b3bf5bfb529b7928b124efe12 Nov 01 00:23:19.350143 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.350071 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4"] Nov 01 00:23:19.351370 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.350784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:19.351370 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.351017 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:19.351370 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.351181 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:19.351370 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.351240 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.351223855 +0000 UTC m=+33.155714302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:19.351370 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.351304 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:19.351370 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.351315 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:19.351370 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.351342 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.351333431 +0000 UTC m=+33.155823867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:19.355524 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.355472 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f6bed4_8a11_4ecb_b334_9f76c5d9ddcc.slice/crio-2eebd2c83e9f1b40e78d7a69756eff17c19860ed63ccbf708465559ec91f82d5 WatchSource:0}: Error finding container 2eebd2c83e9f1b40e78d7a69756eff17c19860ed63ccbf708465559ec91f82d5: Status 404 returned error can't find the container with id 2eebd2c83e9f1b40e78d7a69756eff17c19860ed63ccbf708465559ec91f82d5 Nov 01 00:23:19.369029 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.369012 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gsr76"] Nov 01 00:23:19.375664 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.375641 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e4c8da_dd72_438d_bd5b_a81728051a95.slice/crio-17b3477894b9d5e00f72a358811919763a52c4a9d342b547aad4939a9ffee860 WatchSource:0}: Error finding container 17b3477894b9d5e00f72a358811919763a52c4a9d342b547aad4939a9ffee860: Status 404 returned error can't find the container with id 17b3477894b9d5e00f72a358811919763a52c4a9d342b547aad4939a9ffee860 Nov 01 00:23:19.391693 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.391658 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5"] Nov 01 00:23:19.395113 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.395087 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80541015_4203_4621_bba9_d737aa2f3266.slice/crio-79d68294b96ef2da5fc23df895defd47c35e4ca6f82bc11a21e21cf65baddb4b WatchSource:0}: Error finding container 79d68294b96ef2da5fc23df895defd47c35e4ca6f82bc11a21e21cf65baddb4b: Status 404 returned error can't find the container with id 79d68294b96ef2da5fc23df895defd47c35e4ca6f82bc11a21e21cf65baddb4b Nov 01 00:23:19.415159 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.415137 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7btpl"] Nov 01 00:23:19.417446 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.417425 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b7952d_24a8_40c5_a755_3aeb8269714b.slice/crio-d70a785fe5c03e3777f250930ba2f8965e203e831897847e72e757ec855f9788 WatchSource:0}: Error finding container d70a785fe5c03e3777f250930ba2f8965e203e831897847e72e757ec855f9788: Status 404 returned error can't find the container with id d70a785fe5c03e3777f250930ba2f8965e203e831897847e72e757ec855f9788 Nov 01 00:23:19.438137 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.438106 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb"] Nov 01 00:23:19.442858 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.442831 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783ad43d_a72e_4b58_b403_00eab40f6cdc.slice/crio-cf6725911278feba1375869271551c31483a5aa62d14a4360cb97b8db91fc654 WatchSource:0}: Error finding container cf6725911278feba1375869271551c31483a5aa62d14a4360cb97b8db91fc654: Status 404 returned error can't find the container with id cf6725911278feba1375869271551c31483a5aa62d14a4360cb97b8db91fc654 Nov 01 00:23:19.451687 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.451659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:19.451784 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.451701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:19.451848 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.451790 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:19.451848 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.451802 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:19.451848 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.451847 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.451829971 +0000 UTC m=+33.256320404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:19.452013 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.451872 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.451865183 +0000 UTC m=+33.256355616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:19.452013 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.451906 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:19.452013 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.451942 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:19.452013 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.451976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:19.452013 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.452002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452024 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452037 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452052 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452067 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.452051978 +0000 UTC m=+33.256542412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452091 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.452078587 +0000 UTC m=+33.256569026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452099 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452119 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.452110649 +0000 UTC m=+33.256601110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:19.452307 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:19.452138 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:20.452126907 +0000 UTC m=+33.256617370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:19.470393 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.470333 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nq7f9"] Nov 01 00:23:19.473911 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.473886 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-srh5n"] Nov 01 00:23:19.477189 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.477156 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252a2753_2639_4920_afaf_0fb0ca316d26.slice/crio-577842867deee98b9a5c8ae987761f7eec82d085bca8668087abca837c5a184c WatchSource:0}: Error finding container 577842867deee98b9a5c8ae987761f7eec82d085bca8668087abca837c5a184c: Status 404 returned error can't find the container with id 577842867deee98b9a5c8ae987761f7eec82d085bca8668087abca837c5a184c Nov 01 00:23:19.480176 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.480146 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac18260_596d_49f0_b109_d5750e8c3644.slice/crio-b4dddafc0d945618c75fa2248ac9b8cad882926f88782338b01e9bdef75bedab WatchSource:0}: Error finding container b4dddafc0d945618c75fa2248ac9b8cad882926f88782338b01e9bdef75bedab: Status 404 returned error can't find the container with id b4dddafc0d945618c75fa2248ac9b8cad882926f88782338b01e9bdef75bedab Nov 01 00:23:19.517235 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.517198 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm"] Nov 01 00:23:19.520487 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.520465 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b062870_dd5d_4f35_b400_ea17ebd11878.slice/crio-348d2cd331244881640bfba75d36fde3ce1d8dd8e4961e785deb568be8641486 WatchSource:0}: Error finding container 348d2cd331244881640bfba75d36fde3ce1d8dd8e4961e785deb568be8641486: Status 404 returned error can't find the container with id 348d2cd331244881640bfba75d36fde3ce1d8dd8e4961e785deb568be8641486 Nov 01 00:23:19.541545 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.541508 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hhd5"] Nov 01 00:23:19.544521 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:19.544495 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e97e00_f916_46ea_bcb1_2f0f52ebe2e5.slice/crio-9e5dc3f7f009d302e4ed224a24f80844223df0caf8ee4d91ec321a472b10bf33 WatchSource:0}: Error finding container 9e5dc3f7f009d302e4ed224a24f80844223df0caf8ee4d91ec321a472b10bf33: Status 404 returned error can't find the container with id 9e5dc3f7f009d302e4ed224a24f80844223df0caf8ee4d91ec321a472b10bf33 Nov 01 00:23:19.841376 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.841304 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" event={"ID":"252a2753-2639-4920-afaf-0fb0ca316d26","Type":"ContainerStarted","Data":"577842867deee98b9a5c8ae987761f7eec82d085bca8668087abca837c5a184c"} Nov 01 00:23:19.842618 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.842582 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" event={"ID":"80541015-4203-4621-bba9-d737aa2f3266","Type":"ContainerStarted","Data":"79d68294b96ef2da5fc23df895defd47c35e4ca6f82bc11a21e21cf65baddb4b"} Nov 01 00:23:19.844118 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.844086 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gsr76" event={"ID":"b7e4c8da-dd72-438d-bd5b-a81728051a95","Type":"ContainerStarted","Data":"7d81c1aedba579922ab7c31c7881086bead08b20f3da6df4a5dc63a7a658d663"} Nov 01 00:23:19.844200 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.844133 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gsr76" event={"ID":"b7e4c8da-dd72-438d-bd5b-a81728051a95","Type":"ContainerStarted","Data":"17b3477894b9d5e00f72a358811919763a52c4a9d342b547aad4939a9ffee860"} Nov 01 00:23:19.845291 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.845271 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hhd5" event={"ID":"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5","Type":"ContainerStarted","Data":"9e5dc3f7f009d302e4ed224a24f80844223df0caf8ee4d91ec321a472b10bf33"} Nov 01 00:23:19.846495 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.846468 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" event={"ID":"783ad43d-a72e-4b58-b403-00eab40f6cdc","Type":"ContainerStarted","Data":"cf6725911278feba1375869271551c31483a5aa62d14a4360cb97b8db91fc654"} Nov 01 00:23:19.847957 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.847938 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7btpl" event={"ID":"c5b7952d-24a8-40c5-a755-3aeb8269714b","Type":"ContainerStarted","Data":"c27dcadd477b3a8c2ebbbf514b64a00416ea764ae148ef512d7fad5412d1c353"} Nov 01 00:23:19.847957 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.847967 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7btpl" event={"ID":"c5b7952d-24a8-40c5-a755-3aeb8269714b","Type":"ContainerStarted","Data":"d70a785fe5c03e3777f250930ba2f8965e203e831897847e72e757ec855f9788"} Nov 01 00:23:19.849137 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.849120 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" event={"ID":"70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc","Type":"ContainerStarted","Data":"2eebd2c83e9f1b40e78d7a69756eff17c19860ed63ccbf708465559ec91f82d5"} Nov 01 00:23:19.850256 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.850237 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" event={"ID":"8b062870-dd5d-4f35-b400-ea17ebd11878","Type":"ContainerStarted","Data":"348d2cd331244881640bfba75d36fde3ce1d8dd8e4961e785deb568be8641486"} Nov 01 00:23:19.851396 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.851371 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-srh5n" event={"ID":"0ac18260-596d-49f0-b109-d5750e8c3644","Type":"ContainerStarted","Data":"b4dddafc0d945618c75fa2248ac9b8cad882926f88782338b01e9bdef75bedab"} Nov 01 00:23:19.852834 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.852817 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" event={"ID":"5cfa57d0-764b-42fb-ad36-9ca6dc472766","Type":"ContainerStarted","Data":"2f40004942052bae8110097e31081213824bd8bbf321581ed5bce948fead7d42"} Nov 01 00:23:19.852925 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:19.852844 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" event={"ID":"5cfa57d0-764b-42fb-ad36-9ca6dc472766","Type":"ContainerStarted","Data":"fca98fa50f03b97afd456d5cc8bab5898efb5e4b3bf5bfb529b7928b124efe12"} Nov 01 00:23:20.359905 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.359863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:20.359905 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.359908 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:20.360131 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.360005 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:20.360131 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.360014 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:20.360131 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.360030 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:20.360131 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.360066 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.360051083 +0000 UTC m=+35.164541532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:20.360131 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.360080 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.360074239 +0000 UTC m=+35.164564671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:20.461133 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.461103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:20.461324 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.461147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:20.461324 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.461176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:20.461324 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.461240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:20.461324 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.461265 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:20.461324 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461274 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461339 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.461321971 +0000 UTC m=+35.265812405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461279 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461371 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461377 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.461358983 +0000 UTC m=+35.265849434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461380 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461406 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.461394872 +0000 UTC m=+35.265885331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.461402 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461425 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.461412432 +0000 UTC m=+35.265902882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461379 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461455 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461460 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.461450725 +0000 UTC m=+35.265941175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:20.461518 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:20.461496 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:22.461488763 +0000 UTC m=+35.265979196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:20.770309 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.770245 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gsr76" podStartSLOduration=403.770228842 podStartE2EDuration="6m43.770228842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:20.769612526 +0000 UTC m=+33.574102982" watchObservedRunningTime="2026-11-01 00:23:20.770228842 +0000 UTC m=+33.574719305" Nov 01 00:23:20.887470 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:20.887421 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7btpl" podStartSLOduration=403.887405245 podStartE2EDuration="6m43.887405245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:20.88685341 +0000 UTC m=+33.691343866" watchObservedRunningTime="2026-11-01 00:23:20.887405245 +0000 UTC m=+33.691895696" Nov 01 00:23:21.472813 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:21.472225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:21.472813 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:21.472410 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 00:23:21.472813 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:21.472478 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:23:53.47245955 +0000 UTC m=+66.276950005 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : secret "metrics-daemon-secret" not found Nov 01 00:23:21.573713 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:21.573680 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:21.584339 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:21.584271 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:21.675733 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:21.675015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:21.681725 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:21.681665 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:21.686317 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:21.686287 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:21.711057 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:21.710838 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:22.126452 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.124760 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" podStartSLOduration=405.124737019 podStartE2EDuration="6m45.124737019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:21.004895284 +0000 UTC m=+33.809385740" watchObservedRunningTime="2026-11-01 00:23:22.124737019 +0000 UTC m=+34.929227475" Nov 01 00:23:22.126452 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.126378 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bq86m"] Nov 01 00:23:22.133138 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:22.133043 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579dfa33_9adb_4613_b055_11671039eea2.slice/crio-075828c4fa1dc0f9dfbaac5ebccdd8ca9a8c7198f5f75687ba2c3e4c215a4c68 WatchSource:0}: Error finding container 075828c4fa1dc0f9dfbaac5ebccdd8ca9a8c7198f5f75687ba2c3e4c215a4c68: Status 404 returned error can't find the container with id 075828c4fa1dc0f9dfbaac5ebccdd8ca9a8c7198f5f75687ba2c3e4c215a4c68 Nov 01 00:23:22.370391 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.370147 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m45bc"] Nov 01 00:23:22.374456 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:22.374426 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4454de50_086b_4842_8e00_d2ffae8344d2.slice/crio-c547bac0fa98b6c59c24fa02eedcd524090e44af2fb6a3bc43bc18e3d818b146 WatchSource:0}: Error finding container c547bac0fa98b6c59c24fa02eedcd524090e44af2fb6a3bc43bc18e3d818b146: Status 404 returned error can't find the container with id c547bac0fa98b6c59c24fa02eedcd524090e44af2fb6a3bc43bc18e3d818b146 Nov 01 00:23:22.386057 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.386027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:22.386169 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.386076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:22.386225 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.386172 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:22.386278 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.386221 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:22.386278 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.386237 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:22.386278 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.386247 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.38622794 +0000 UTC m=+39.190718393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:22.386278 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.386275 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.386264017 +0000 UTC m=+39.190754454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:22.487178 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.487127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:22.487353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.487190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:22.487353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.487274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:22.487353 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.487314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:22.487353 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487324 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:22.487565 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487399 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.48737937 +0000 UTC m=+39.291869819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:22.487565 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.487425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:22.487565 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487470 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:22.487565 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.487479 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:22.487565 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487524 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.487509327 +0000 UTC m=+39.291999763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487595 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.487584366 +0000 UTC m=+39.292074815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487672 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487706 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.487695712 +0000 UTC m=+39.292186186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487667 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487735 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487772 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.487758207 +0000 UTC m=+39.292248655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487770 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:22.487924 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:22.487820 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:26.487808999 +0000 UTC m=+39.292299448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:22.872602 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.872561 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m45bc" event={"ID":"4454de50-086b-4842-8e00-d2ffae8344d2","Type":"ContainerStarted","Data":"c547bac0fa98b6c59c24fa02eedcd524090e44af2fb6a3bc43bc18e3d818b146"} Nov 01 00:23:22.874411 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.874354 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bq86m" event={"ID":"579dfa33-9adb-4613-b055-11671039eea2","Type":"ContainerStarted","Data":"075828c4fa1dc0f9dfbaac5ebccdd8ca9a8c7198f5f75687ba2c3e4c215a4c68"} Nov 01 00:23:22.877582 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:22.876860 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" event={"ID":"70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc","Type":"ContainerStarted","Data":"74df471fa849f2f38946bd54bb25a290d1f30f4c142b886a8ce1e0a1ce722906"} Nov 01 00:23:24.887729 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:24.886693 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/0.log" Nov 01 00:23:24.887729 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:24.886736 2570 generic.go:356] "Generic (PLEG): container finished" podID="252a2753-2639-4920-afaf-0fb0ca316d26" containerID="34e706b1d361cca325f29001fed2a3be4eb665f51acec358685f9a95dfe5ddde" exitCode=255 Nov 01 00:23:24.887729 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:24.886790 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" event={"ID":"252a2753-2639-4920-afaf-0fb0ca316d26","Type":"ContainerDied","Data":"34e706b1d361cca325f29001fed2a3be4eb665f51acec358685f9a95dfe5ddde"} Nov 01 00:23:24.887729 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:24.887165 2570 scope.go:122] "RemoveContainer" containerID="34e706b1d361cca325f29001fed2a3be4eb665f51acec358685f9a95dfe5ddde" Nov 01 00:23:25.005228 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.005164 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" podStartSLOduration=359.408974258 podStartE2EDuration="6m2.005144402s" totalImagesPullingTime="2.596170144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:22.992545507 +0000 UTC m=+35.797035964" watchObservedRunningTime="2026-11-01 00:23:25.005144402 +0000 UTC m=+37.809634858" Nov 01 00:23:25.905178 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.905141 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" event={"ID":"783ad43d-a72e-4b58-b403-00eab40f6cdc","Type":"ContainerStarted","Data":"fcae44f4cf72e9e0d532fc74152e4ed221c563a4251f5fabcd63eb88c28075ba"} Nov 01 00:23:25.907386 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.907358 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-srh5n" event={"ID":"0ac18260-596d-49f0-b109-d5750e8c3644","Type":"ContainerStarted","Data":"2ed4996273be3b24a329987ea56c8e86bf346535c294c4ca58f85096772af5a0"} Nov 01 00:23:25.915072 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.911201 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/1.log" Nov 01 00:23:25.915072 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.912879 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/0.log" Nov 01 00:23:25.915072 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.912916 2570 generic.go:356] "Generic (PLEG): container finished" podID="252a2753-2639-4920-afaf-0fb0ca316d26" containerID="1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd" exitCode=255 Nov 01 00:23:25.915072 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.913000 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" event={"ID":"252a2753-2639-4920-afaf-0fb0ca316d26","Type":"ContainerDied","Data":"1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd"} Nov 01 00:23:25.915072 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.913047 2570 scope.go:122] "RemoveContainer" containerID="34e706b1d361cca325f29001fed2a3be4eb665f51acec358685f9a95dfe5ddde" Nov 01 00:23:25.915072 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.913417 2570 scope.go:122] "RemoveContainer" containerID="1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd" Nov 01 00:23:25.915072 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:25.913659 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq7f9_openshift-console-operator(252a2753-2639-4920-afaf-0fb0ca316d26)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" podUID="252a2753-2639-4920-afaf-0fb0ca316d26" Nov 01 00:23:25.921034 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:25.921005 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" event={"ID":"80541015-4203-4621-bba9-d737aa2f3266","Type":"ContainerStarted","Data":"b3d4bfd1b16cae4582f062644bfbfc307deb92d7446a87683e3559696ca14f1d"} Nov 01 00:23:26.149714 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.142286 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" podStartSLOduration=352.149977822 podStartE2EDuration="5m58.142268421s" totalImagesPullingTime="5.992290599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:26.025903656 +0000 UTC m=+38.830394115" watchObservedRunningTime="2026-11-01 00:23:26.142268421 +0000 UTC m=+38.946758881" Nov 01 00:23:26.261391 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.261321 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" podStartSLOduration=355.199973617 podStartE2EDuration="6m1.261303437s" totalImagesPullingTime="6.06132982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:26.14378393 +0000 UTC m=+38.948274386" watchObservedRunningTime="2026-11-01 00:23:26.261303437 +0000 UTC m=+39.065793892" Nov 01 00:23:26.381143 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.379830 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-srh5n" podStartSLOduration=350.424974604 podStartE2EDuration="5m56.379813655s" totalImagesPullingTime="5.954839051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:26.379481031 +0000 UTC m=+39.183971487" watchObservedRunningTime="2026-11-01 00:23:26.379813655 +0000 UTC m=+39.184304109" Nov 01 00:23:26.432018 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.431932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:26.432018 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.431981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:26.432229 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.432103 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:26.432229 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.432112 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:26.432229 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.432125 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:26.432229 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.432179 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.432162286 +0000 UTC m=+47.236652741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:26.432229 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.432195 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.432187865 +0000 UTC m=+47.236678311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:26.533264 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.533222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:26.533445 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.533283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:26.533445 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.533356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:26.533445 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.533391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:26.533445 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.533432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:26.533696 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.533469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:26.533696 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.533655 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.533610184 +0000 UTC m=+47.338100633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:26.533818 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.533751 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:26.533818 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.533788 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.533777182 +0000 UTC m=+47.338267621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:26.533933 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.533843 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:26.533933 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.533871 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.533862735 +0000 UTC m=+47.338353173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:26.533933 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.533925 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:26.534082 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.533953 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.533943628 +0000 UTC m=+47.338434075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:26.534082 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.534013 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:26.534082 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.534023 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:26.534082 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.534050 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.534041373 +0000 UTC m=+47.338531809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:26.534236 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.534104 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:26.534236 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.534129 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:34.534120687 +0000 UTC m=+47.338611120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:26.857836 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.856309 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k2s97"] Nov 01 00:23:26.869235 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.869212 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:26.918250 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.918219 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k2s97"] Nov 01 00:23:26.922126 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.922100 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 00:23:26.922263 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.922231 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqdjj\"" Nov 01 00:23:26.928338 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.928312 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hhd5" event={"ID":"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5","Type":"ContainerStarted","Data":"3160af0fd7a1168e7ae6372dbeda7cc9da492248848a4f9a552612f700caaad3"} Nov 01 00:23:26.931582 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.931555 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" event={"ID":"8b062870-dd5d-4f35-b400-ea17ebd11878","Type":"ContainerStarted","Data":"dc7d6b75a8d95a0c6e9cfa1123fd99c0d7217e9527f00da0354337192f39ce00"} Nov 01 00:23:26.934054 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.934035 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/1.log" Nov 01 00:23:26.934878 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.934859 2570 scope.go:122] "RemoveContainer" containerID="1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd" Nov 01 00:23:26.935055 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:26.935010 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq7f9_openshift-console-operator(252a2753-2639-4920-afaf-0fb0ca316d26)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" podUID="252a2753-2639-4920-afaf-0fb0ca316d26" Nov 01 00:23:26.938471 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.938440 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:26.938581 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.938560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3246fb55-62ae-4d36-a951-75206d4f99d1-crio-socket\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:26.938690 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.938643 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3246fb55-62ae-4d36-a951-75206d4f99d1-data-volume\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:26.938751 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.938703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5twnl\" (UniqueName: \"kubernetes.io/projected/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-api-access-5twnl\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:26.938751 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.938744 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:26.986584 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:26.986483 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 00:23:27.039963 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.039930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.040146 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.040019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3246fb55-62ae-4d36-a951-75206d4f99d1-crio-socket\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.040146 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.040089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3246fb55-62ae-4d36-a951-75206d4f99d1-data-volume\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.041905 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.040180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5twnl\" (UniqueName: \"kubernetes.io/projected/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-api-access-5twnl\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.041905 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.040266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.041905 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.040680 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 00:23:27.041905 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.040739 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:27.540721798 +0000 UTC m=+40.345212249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : secret "insights-runtime-extractor-tls" not found Nov 01 00:23:27.041905 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.041271 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3246fb55-62ae-4d36-a951-75206d4f99d1-data-volume\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.041905 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.041775 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 00:23:27.041905 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.041826 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:27.541810768 +0000 UTC m=+40.346301220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : configmap "kube-rbac-proxy" not found Nov 01 00:23:27.042263 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.042228 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3246fb55-62ae-4d36-a951-75206d4f99d1-crio-socket\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.120345 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.120306 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5twnl\" (UniqueName: \"kubernetes.io/projected/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-api-access-5twnl\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.227840 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.227774 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" podStartSLOduration=352.04948149 podStartE2EDuration="5m59.227756106s" totalImagesPullingTime="7.178274616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:27.225683343 +0000 UTC m=+40.030173802" watchObservedRunningTime="2026-11-01 00:23:27.227756106 +0000 UTC m=+40.032246616" Nov 01 00:23:27.343267 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.342125 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5hhd5" podStartSLOduration=3.032114505 podStartE2EDuration="9.342103696s" totalImagesPullingTime="6.309989191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:27.340797441 +0000 UTC m=+40.145287896" watchObservedRunningTime="2026-11-01 00:23:27.342103696 +0000 UTC m=+40.146594154" Nov 01 00:23:27.545757 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.544878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.545757 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.545026 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:27.545757 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.545190 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 00:23:27.545757 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.545255 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:28.54523623 +0000 UTC m=+41.349726666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : secret "insights-runtime-extractor-tls" not found Nov 01 00:23:27.545757 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.545656 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 00:23:27.545757 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:27.545710 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:28.545693994 +0000 UTC m=+41.350184431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : configmap "kube-rbac-proxy" not found Nov 01 00:23:27.733336 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.733262 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5"] Nov 01 00:23:27.752072 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.752040 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" Nov 01 00:23:27.794916 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.794833 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5"] Nov 01 00:23:27.804095 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.804064 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 00:23:27.804258 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.804115 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:27.804258 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.804191 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2ck8t\"" Nov 01 00:23:27.847936 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.847850 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsgbt\" (UniqueName: \"kubernetes.io/projected/3b775de6-474a-4551-8ddc-ade7a8c25fab-kube-api-access-hsgbt\") pod \"migrator-66df5d496b-6dzb5\" (UID: \"3b775de6-474a-4551-8ddc-ade7a8c25fab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" Nov 01 00:23:27.948884 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:27.948857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgbt\" (UniqueName: \"kubernetes.io/projected/3b775de6-474a-4551-8ddc-ade7a8c25fab-kube-api-access-hsgbt\") pod \"migrator-66df5d496b-6dzb5\" (UID: \"3b775de6-474a-4551-8ddc-ade7a8c25fab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" Nov 01 00:23:28.009965 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.009937 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgbt\" (UniqueName: \"kubernetes.io/projected/3b775de6-474a-4551-8ddc-ade7a8c25fab-kube-api-access-hsgbt\") pod \"migrator-66df5d496b-6dzb5\" (UID: \"3b775de6-474a-4551-8ddc-ade7a8c25fab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" Nov 01 00:23:28.082606 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.082573 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" Nov 01 00:23:28.478489 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.478390 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5"] Nov 01 00:23:28.481446 ip-10-0-137-150 kubenswrapper[2570]: W1101 00:23:28.481405 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b775de6_474a_4551_8ddc_ade7a8c25fab.slice/crio-efcfa88b3b73200463e02d198abe5e0182425c7e44b154aaaca2c257f8ab8f23 WatchSource:0}: Error finding container efcfa88b3b73200463e02d198abe5e0182425c7e44b154aaaca2c257f8ab8f23: Status 404 returned error can't find the container with id efcfa88b3b73200463e02d198abe5e0182425c7e44b154aaaca2c257f8ab8f23 Nov 01 00:23:28.555759 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.555718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:28.555951 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:28.555831 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 00:23:28.555951 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.555847 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:28.555951 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:28.555915 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:30.555893764 +0000 UTC m=+43.360384209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : configmap "kube-rbac-proxy" not found Nov 01 00:23:28.556125 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:28.555986 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 00:23:28.556125 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:28.556039 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:30.556026536 +0000 UTC m=+43.360516974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : secret "insights-runtime-extractor-tls" not found Nov 01 00:23:28.943286 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.943248 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bq86m" event={"ID":"579dfa33-9adb-4613-b055-11671039eea2","Type":"ContainerStarted","Data":"af5070880b24d41a447f2b8c48065eee2be467baf226cebe77d738a5600091f3"} Nov 01 00:23:28.943491 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.943353 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:28.944505 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:28.944472 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" event={"ID":"3b775de6-474a-4551-8ddc-ade7a8c25fab","Type":"ContainerStarted","Data":"efcfa88b3b73200463e02d198abe5e0182425c7e44b154aaaca2c257f8ab8f23"} Nov 01 00:23:29.052527 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:29.052496 2570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:29.052971 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:29.052558 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:29.053030 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:29.052967 2570 scope.go:122] "RemoveContainer" containerID="1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd" Nov 01 00:23:29.053211 ip-10-0-137-150 kubenswrapper[2570]: E1101 00:23:29.053186 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq7f9_openshift-console-operator(252a2753-2639-4920-afaf-0fb0ca316d26)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" podUID="252a2753-2639-4920-afaf-0fb0ca316d26" Nov 01 00:23:29.058902 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:29.058849 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bq86m" podStartSLOduration=34.192987496 podStartE2EDuration="40.058832782s" totalImagesPullingTime="5.865845286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:23:29.058427654 +0000 UTC m=+41.862918110" watchObservedRunningTime="2026-11-01 00:23:29.058832782 +0000 UTC m=+41.863323246" Nov 01 00:23:29.948861 ip-10-0-137-150 kubenswrapper[2570]: I1101 00:23:29.948776 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m45bc" event={"ID":"4454de50-086b-4842-8e00-d2ffae8344d2","Type":"ContainerStarted","Data":"7fcec3aa2a07b4e0612b47da95130740d0f1127f9fcb13614ab4f72a46c07214"} Nov 01 00:23:29.961962 ip-10-0-137-150 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 00:23:29.976177 ip-10-0-137-150 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 00:23:29.976347 ip-10-0-137-150 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 00:23:29.976399 ip-10-0-137-150 systemd[1]: kubelet.service: Consumed 3.061s CPU time. Nov 01 00:23:29.986151 ip-10-0-137-150 systemd[1]: Starting Kubernetes Kubelet... Nov 01 00:23:30.082531 ip-10-0-137-150 kubenswrapper[4910]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:23:30.082531 ip-10-0-137-150 kubenswrapper[4910]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 00:23:30.082531 ip-10-0-137-150 kubenswrapper[4910]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:23:30.082531 ip-10-0-137-150 kubenswrapper[4910]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 00:23:30.085110 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.084996 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.085110 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085104 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.085110 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085109 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.085110 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085112 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.085110 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085115 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.085110 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085118 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085121 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085124 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085126 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085129 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085132 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085135 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085137 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085140 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085143 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085145 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085148 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085150 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085153 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085156 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085159 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085162 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085164 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085167 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085170 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.085377 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085172 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085177 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085185 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085195 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085200 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085203 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085206 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085210 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085214 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085220 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085224 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085227 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085232 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085236 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085241 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085245 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085250 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085254 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085258 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.085994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085261 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085263 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085266 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085270 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085273 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085276 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085279 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085281 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085284 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085287 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085292 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085296 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085301 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085306 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085309 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085312 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085315 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085318 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085321 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.086554 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085323 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085326 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085329 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085331 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085334 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085337 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085339 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085342 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085344 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085347 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085349 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085352 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085355 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085358 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085361 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085363 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085366 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085369 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085372 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.087134 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085375 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085377 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085380 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085383 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085385 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085411 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085417 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085422 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085425 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085427 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085430 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085434 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085440 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085444 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085449 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085454 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085458 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085462 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085468 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085473 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.087827 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085477 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085481 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085485 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.085489 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086385 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086393 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086396 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086399 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086401 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086404 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086407 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086410 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086413 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086415 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086418 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086421 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086423 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086427 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086429 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086432 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.088524 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086434 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086437 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086440 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086443 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086445 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086448 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086451 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086453 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086456 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086458 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086461 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086463 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086466 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086469 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086471 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086473 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086476 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086478 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086481 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086485 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.089155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086489 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086494 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086498 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086502 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086505 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086527 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086530 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086547 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086550 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086553 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086556 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086567 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086570 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086573 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086576 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086578 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086581 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086583 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086586 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086589 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.089848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086592 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086596 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086600 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086604 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086609 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086614 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086636 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086642 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086646 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086650 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086655 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086660 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086664 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086669 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086673 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086677 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086681 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086686 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086690 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086694 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.090601 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086699 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086704 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086708 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086713 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086720 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086727 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086731 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086734 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086737 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086740 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086742 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086745 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086747 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086750 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086753 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086755 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086758 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086762 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086766 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.091276 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086771 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086778 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086783 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086789 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086795 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086799 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086803 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086807 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086811 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086816 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.086820 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087414 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087422 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087426 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087429 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087432 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087435 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087438 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087442 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.091875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087447 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087451 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087456 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087459 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087461 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087464 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087467 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087469 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087472 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087475 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087478 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087480 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087483 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087485 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087488 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087490 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087493 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087496 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087498 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.092510 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087501 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087505 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087509 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087513 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087518 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087525 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087530 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087534 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087538 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087543 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087549 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087553 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087557 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087561 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087566 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087571 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087575 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087578 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087581 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.093139 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087583 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087586 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087589 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087592 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087596 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087600 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087604 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087608 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087613 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087616 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087634 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087639 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087643 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087647 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087651 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087654 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087657 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087659 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087662 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087667 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.093797 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087671 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087675 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087680 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087684 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087688 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087693 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087697 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087702 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087707 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087711 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087715 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087719 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087723 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087728 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087733 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087737 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087741 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087745 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087750 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087755 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.094511 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087760 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087763 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087765 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087769 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087773 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087777 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087782 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087786 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087790 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087794 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087797 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087802 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087806 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087811 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087815 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087819 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087822 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087825 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087830 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.087834 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.095187 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.087965 4910 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.087982 4910 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.087991 4910 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.087998 4910 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088005 4910 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088011 4910 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088015 4910 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088019 4910 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088022 4910 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088025 4910 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088029 4910 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088032 4910 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088037 4910 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088040 4910 flags.go:64] FLAG: --cgroup-root="" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088042 4910 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088047 4910 flags.go:64] FLAG: --client-ca-file="" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088050 4910 flags.go:64] FLAG: --cloud-provider="external" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088053 4910 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088058 4910 flags.go:64] FLAG: --cluster-domain="" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088061 4910 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088064 4910 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088069 4910 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088075 4910 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088081 4910 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 00:23:30.095791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088087 4910 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088092 4910 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088096 4910 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088099 4910 flags.go:64] FLAG: --contention-profiling="false" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088102 4910 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088105 4910 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088108 4910 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088111 4910 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088115 4910 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088118 4910 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088121 4910 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088124 4910 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088127 4910 flags.go:64] FLAG: --enable-server="true" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088130 4910 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088134 4910 flags.go:64] FLAG: --event-burst="100" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088137 4910 flags.go:64] FLAG: --event-qps="50" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088140 4910 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088143 4910 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088146 4910 flags.go:64] FLAG: --eviction-hard="" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088151 4910 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088155 4910 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088158 4910 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088161 4910 flags.go:64] FLAG: --eviction-soft="" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088168 4910 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088173 4910 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 00:23:30.096638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088178 4910 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088183 4910 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088188 4910 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088191 4910 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088194 4910 flags.go:64] FLAG: --feature-gates="" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088198 4910 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088201 4910 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088204 4910 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088207 4910 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088212 4910 flags.go:64] FLAG: --healthz-port="10248" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088215 4910 flags.go:64] FLAG: --help="false" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088218 4910 flags.go:64] FLAG: --hostname-override="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088222 4910 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088226 4910 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088231 4910 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088237 4910 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088243 4910 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088247 4910 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088252 4910 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088257 4910 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088261 4910 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088266 4910 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088270 4910 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088275 4910 flags.go:64] FLAG: --kube-reserved="" Nov 01 00:23:30.097514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088280 4910 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088285 4910 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088290 4910 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088296 4910 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088299 4910 flags.go:64] FLAG: --lock-file="" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088301 4910 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088305 4910 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088308 4910 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088333 4910 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088337 4910 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088343 4910 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088348 4910 flags.go:64] FLAG: --logging-format="text" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088352 4910 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088357 4910 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088363 4910 flags.go:64] FLAG: --manifest-url="" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088368 4910 flags.go:64] FLAG: --manifest-url-header="" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088375 4910 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088381 4910 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088386 4910 flags.go:64] FLAG: --max-pods="110" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088390 4910 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088393 4910 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088395 4910 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088398 4910 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088401 4910 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088404 4910 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 00:23:30.098364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088407 4910 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088415 4910 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088418 4910 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088421 4910 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088424 4910 flags.go:64] FLAG: --pod-cidr="" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088427 4910 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088430 4910 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088435 4910 flags.go:64] FLAG: --pods-per-core="0" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088440 4910 flags.go:64] FLAG: --port="10250" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088445 4910 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088453 4910 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-076faa3b1608e5546" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088459 4910 flags.go:64] FLAG: --qos-reserved="" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088464 4910 flags.go:64] FLAG: --read-only-port="10255" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088471 4910 flags.go:64] FLAG: --register-node="true" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088474 4910 flags.go:64] FLAG: --register-with-taints="" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088478 4910 flags.go:64] FLAG: --registry-burst="10" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088482 4910 flags.go:64] FLAG: --registry-qps="5" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088487 4910 flags.go:64] FLAG: --reserved-cpus="" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088491 4910 flags.go:64] FLAG: --reserved-memory="" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088498 4910 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088503 4910 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088509 4910 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088514 4910 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088519 4910 flags.go:64] FLAG: --runonce="false" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088524 4910 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088529 4910 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088536 4910 flags.go:64] FLAG: --seccomp-default="false" Nov 01 00:23:30.099317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088541 4910 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088545 4910 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088550 4910 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088554 4910 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088559 4910 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088563 4910 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088568 4910 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088572 4910 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088577 4910 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088582 4910 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088586 4910 flags.go:64] FLAG: --system-cgroups="" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088591 4910 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088600 4910 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088604 4910 flags.go:64] FLAG: --tls-cert-file="" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088609 4910 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088617 4910 flags.go:64] FLAG: --tls-min-version="" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088636 4910 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088640 4910 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088647 4910 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088652 4910 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088656 4910 flags.go:64] FLAG: --v="2" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088660 4910 flags.go:64] FLAG: --version="false" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088665 4910 flags.go:64] FLAG: --vmodule="" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088668 4910 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.088672 4910 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 00:23:30.100492 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088862 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088871 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088876 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088881 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088886 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088890 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088893 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088896 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088899 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088902 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088907 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088912 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088917 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088921 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088926 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088930 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088933 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088936 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088938 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088941 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.101509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088944 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088948 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088955 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088958 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088963 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088970 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088974 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088979 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088983 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088988 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088993 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.088997 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089001 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089005 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089007 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089011 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089015 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089020 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089025 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.102267 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089030 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089035 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089038 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089041 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089043 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089046 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089049 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089052 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089054 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089057 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089059 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089062 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089065 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089069 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089074 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089080 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089085 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089089 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089097 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.102996 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089103 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089107 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089111 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089113 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089116 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089120 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089124 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089129 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089133 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089137 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089142 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089146 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089148 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089151 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089154 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089157 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089160 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089163 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089165 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089168 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.103588 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089171 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089175 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089178 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089183 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089187 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089192 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089197 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089202 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089208 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089213 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089217 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089221 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089224 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089227 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089229 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089232 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089234 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089237 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089240 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089242 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.104269 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089245 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089247 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089252 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089255 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089258 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089261 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089263 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089266 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.089302 4910 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089352 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089357 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089362 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089366 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089371 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089376 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089380 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.104988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089385 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089390 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089395 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089399 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089402 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089405 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089408 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089411 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089414 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089417 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089419 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089422 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089426 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089430 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089434 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089438 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089442 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089446 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089449 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089452 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.105502 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089454 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089457 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089459 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089462 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089465 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089469 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089471 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089474 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089476 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089479 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089481 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089484 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089486 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089489 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089491 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089495 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089499 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089503 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089507 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089512 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.106191 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089518 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089522 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089525 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089528 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089530 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089533 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089536 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089541 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089545 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089549 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089553 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089560 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089565 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089570 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089574 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089578 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089582 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089585 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089590 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.106856 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089595 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089597 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089600 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089603 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089606 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089610 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089614 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089633 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089638 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089642 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089647 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089649 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089652 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089655 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089658 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089661 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089663 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089666 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089670 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.107459 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089676 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089682 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089686 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089691 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089694 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089697 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089699 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089703 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089706 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089708 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089711 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089713 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089718 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089721 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089723 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089726 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089729 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089731 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089733 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089745 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.108126 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.089749 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.094411 4910 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.094429 4910 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.094451 4910 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.094458 4910 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094542 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094551 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094557 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094563 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094568 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094573 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094577 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094582 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094586 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094591 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094596 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094600 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094604 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094609 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.108890 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094613 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094617 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094641 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094646 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094651 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094655 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094661 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094665 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094670 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094674 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094679 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094683 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094688 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094692 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094697 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094701 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094706 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094710 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094714 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.109544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094719 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094723 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094728 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094732 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094737 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094741 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094746 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094750 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094754 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094758 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094765 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094770 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094775 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094780 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094786 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094792 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094798 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094803 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094808 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.110155 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094812 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094818 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094823 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094828 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094832 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094837 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094841 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094846 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094851 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094854 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094859 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094863 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094868 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094872 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094878 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094883 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094888 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094893 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094898 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094902 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.110777 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094907 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094913 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094918 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094922 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094926 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094932 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094937 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094942 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094947 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094952 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094957 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094962 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094967 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094971 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094977 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094982 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094986 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094990 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.094995 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.111387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095000 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095004 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095009 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095014 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095018 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095020 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095023 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095026 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095028 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095031 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095034 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095036 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095038 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095041 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095044 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.095085 4910 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:23:30.111988 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095261 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095269 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095272 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095275 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095278 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095280 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095283 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095285 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095288 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095290 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095293 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095295 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095299 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095302 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095304 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095307 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095309 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095312 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095314 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095316 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.112507 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095319 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095323 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095327 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095330 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095333 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095336 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095339 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095341 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095343 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095346 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095349 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095351 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095354 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095356 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095358 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095361 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095363 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095366 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095370 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095374 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.113221 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095377 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095381 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095385 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095388 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095390 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095394 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095397 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095399 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095402 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095404 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095406 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095409 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095411 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095414 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095416 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095418 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095421 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095423 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095425 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095428 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.113931 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095430 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095435 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095438 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095440 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095443 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095445 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095448 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095450 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095453 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095455 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095458 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095461 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095464 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095466 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095469 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095471 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095474 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095477 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095480 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.114544 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095482 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095485 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095488 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095490 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095493 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095495 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095497 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095500 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095502 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095505 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095507 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095509 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095513 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095517 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095523 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095528 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095532 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095536 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095541 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.115275 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095543 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095546 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095548 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095551 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095553 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095556 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095559 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095561 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.095586 4910 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095657 4910 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095663 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095666 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095668 4910 feature_gate.go:386] unrecognized feature gate: Example Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095673 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095676 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095679 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 00:23:30.116099 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095681 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095684 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095687 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095691 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095695 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095699 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095703 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095707 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095711 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095716 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095719 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095724 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095726 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095729 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095731 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095734 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095736 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095740 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095744 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 00:23:30.116604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095747 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095749 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095752 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095754 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095756 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095759 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095761 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095764 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095766 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095768 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095771 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095774 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095776 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095779 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095781 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095785 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095788 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095791 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095793 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 00:23:30.117294 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095796 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095798 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095801 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095804 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095806 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095810 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095813 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095815 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095819 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095821 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095824 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095826 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095829 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095831 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095834 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095838 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095841 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095843 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095846 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 00:23:30.117994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095848 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095851 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095853 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095856 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095860 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095864 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095868 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095871 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095873 4910 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095877 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095880 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095883 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095886 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095888 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095891 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095893 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095896 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095898 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095902 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095905 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095907 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 00:23:30.118789 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095910 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095913 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095915 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095917 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095920 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095923 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095925 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095928 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095930 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095933 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095935 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095937 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095940 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095942 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095945 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095948 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095951 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095953 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095956 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 00:23:30.119722 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095959 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:30.095963 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.096031 4910 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.097167 4910 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.097234 4910 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.097442 4910 server.go:1014] "Starting client certificate rotation" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.097591 4910 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.097653 4910 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 00:17:48 +0000 UTC" deadline="2028-04-12 22:39:18.096153056 +0000 UTC" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.097697 4910 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12694h15m47.998459064s" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.098007 4910 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.099306 4910 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.102063 4910 log.go:25] "Validated CRI v1 runtime API" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.104173 4910 log.go:25] "Validated CRI v1 image API" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.104997 4910 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 00:23:30.120395 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.107221 4910 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 814c003a-fcc6-40dc-abcc-4dffac607da7:/dev/nvme0n1p4 85d8d246-5371-45b5-8fc8-aabfb348aa5f:/dev/nvme0n1p3] Nov 01 00:23:30.120913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.107244 4910 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05423de021baa9deb6552031a4ebf17180a97b8365774aada80623b416242df3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05423de021baa9deb6552031a4ebf17180a97b8365774aada80623b416242df3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/075828c4fa1dc0f9dfbaac5ebccdd8ca9a8c7198f5f75687ba2c3e4c215a4c68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/075828c4fa1dc0f9dfbaac5ebccdd8ca9a8c7198f5f75687ba2c3e4c215a4c68/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17b3477894b9d5e00f72a358811919763a52c4a9d342b547aad4939a9ffee860/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17b3477894b9d5e00f72a358811919763a52c4a9d342b547aad4939a9ffee860/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2eebd2c83e9f1b40e78d7a69756eff17c19860ed63ccbf708465559ec91f82d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2eebd2c83e9f1b40e78d7a69756eff17c19860ed63ccbf708465559ec91f82d5/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/348d2cd331244881640bfba75d36fde3ce1d8dd8e4961e785deb568be8641486/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/348d2cd331244881640bfba75d36fde3ce1d8dd8e4961e785deb568be8641486/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a052aef8177eb60c0002f77bf693a10077a9e50abe7de0f21054ba97cd8fd68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a052aef8177eb60c0002f77bf693a10077a9e50abe7de0f21054ba97cd8fd68/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/577842867deee98b9a5c8ae987761f7eec82d085bca8668087abca837c5a184c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/577842867deee98b9a5c8ae987761f7eec82d085bca8668087abca837c5a184c/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f765550379f004449a7a7c7498e53f43ee64939579ef6aeca79bd25b17f5aeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f765550379f004449a7a7c7498e53f43ee64939579ef6aeca79bd25b17f5aeb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79d68294b96ef2da5fc23df895defd47c35e4ca6f82bc11a21e21cf65baddb4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79d68294b96ef2da5fc23df895defd47c35e4ca6f82bc11a21e21cf65baddb4b/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80fda66eb807be3c9b964db1d67e0ad1ecbeb3ce6d6648b82c2567d4909e5d05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80fda66eb807be3c9b964db1d67e0ad1ecbeb3ce6d6648b82c2567d4909e5d05/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e5dc3f7f009d302e4ed224a24f80844223df0caf8ee4d91ec321a472b10bf33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e5dc3f7f009d302e4ed224a24f80844223df0caf8ee4d91ec321a472b10bf33/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4dddafc0d945618c75fa2248ac9b8cad882926f88782338b01e9bdef75bedab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4dddafc0d945618c75fa2248ac9b8cad882926f88782338b01e9bdef75bedab/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b734bc2834027f0b0318f25842ed43adebcb55f86afe9034a00589c6ae01a414/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b734bc2834027f0b0318f25842ed43adebcb55f86afe9034a00589c6ae01a414/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c547bac0fa98b6c59c24fa02eedcd524090e44af2fb6a3bc43bc18e3d818b146/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c547bac0fa98b6c59c24fa02eedcd524090e44af2fb6a3bc43bc18e3d818b146/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf6725911278feba1375869271551c31483a5aa62d14a4360cb97b8db91fc654/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf6725911278feba1375869271551c31483a5aa62d14a4360cb97b8db91fc654/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2375a2b5cd32d8cd8837b3ac14dc0910d62d65c1d80caaf0ef32c22accefaa7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2375a2b5cd32d8cd8837b3ac14dc0910d62d65c1d80caaf0ef32c22accefaa7/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d70a785fe5c03e3777f250930ba2f8965e203e831897847e72e757ec855f9788/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d70a785fe5c03e3777f250930ba2f8965e203e831897847e72e757ec855f9788/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df5599d793d24509604a61bebdc994ce74c1c9d926a7a3da161dfed6d4ac326f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df5599d793d24509604a61bebdc994ce74c1c9d926a7a3da161dfed6d4ac326f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efcfa88b3b73200463e02d198abe5e0182425c7e44b154aaaca2c257f8ab8f23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efcfa88b3b73200463e02d198abe5e0182425c7e44b154aaaca2c257f8ab8f23/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f99bf252830c0acab95b3f4507930a31d6bbf9858daa9264855613fa09492337/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f99bf252830c0acab95b3f4507930a31d6bbf9858daa9264855613fa09492337/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fca98fa50f03b97afd456d5cc8bab5898efb5e4b3bf5bfb529b7928b124efe12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fca98fa50f03b97afd456d5cc8bab5898efb5e4b3bf5bfb529b7928b124efe12/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~projected/kube-api-access-dwpgz:{mountpoint:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~projected/kube-api-access-dwpgz major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ac18260-596d-49f0-b109-d5750e8c3644/volumes/kubernetes.io~projected/kube-api-access-8k7vb:{mountpoint:/var/lib/kubelet/pods/0ac18260-596d-49f0-b109-d5750e8c3644/volumes/kubernetes.io~projected/kube-api-access-8k7vb major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ac18260-596d-49f0-b109-d5750e8c3644/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0ac18260-596d-49f0-b109-d5750e8c3644/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~projected/kube-api-access-lsw6b:{mountpoint:/var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~projected/kube-api-access-lsw6b major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~secret/default-certificate major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~secret/stats-auth major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7/volumes/kubernetes.io~projected/kube-api-access-dh8m6:{mountpoint:/var/lib/kubelet/pods/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7/volumes/kubernetes.io~projected/kube-api-access-dh8m6 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2508f7ff-f9da-48fe-8e1c-ba02568e0899/volumes/kubernetes.io~projected/kube-api-access-nngm4:{mountpoint:/var/lib/kubelet/pods/2508f7ff-f9da-48fe-8e1c-ba02568e0899/volumes/kubernetes.io~projected/kube-api-access-nngm4 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/252a2753-2639-4920-afaf-0fb0ca316d26/volumes/kubernetes.io~projected/kube-api-access-wghrw:{mountpoint:/var/lib/kubelet/pods/252a2753-2639-4920-afaf-0fb0ca316d26/volumes/kubernetes.io~projected/kube-api-access-wghrw major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/252a2753-2639-4920-afaf-0fb0ca316d26/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/252a2753-2639-4920-afaf-0fb0ca316d26/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3246fb55-62ae-4d36-a951-75206d4f99d1/volumes/kubernetes.io~projected/kube-api-access-5twnl:{mountpoint:/var/lib/kubelet/pods/3246fb55-62ae-4d36-a951-75206d4f99d1/volumes/kubernetes.io~projected/kube-api-access-5twnl major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38761a0d-2c71-4f3b-8d5d-39cdf321763f/volumes/kubernetes.io~projected/kube-api-access-2j6l2:{mountpoint:/var/lib/kubelet/pods/38761a0d-2c71-4f3b-8d5d-39cdf321763f/volumes/kubernetes.io~projected/kube-api-access-2j6l2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b775de6-474a-4551-8ddc-ade7a8c25fab/volumes/kubernetes.io~projected/kube-api-access-hsgbt:{mountpoint:/var/lib/kubelet/pods/3b775de6-474a-4551-8ddc-ade7a8c25fab/volumes/kubernetes.io~projected/kube-api-access-hsgbt major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4454de50-086b-4842-8e00-d2ffae8344d2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4454de50-086b-4842-8e00-d2ffae8344d2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~projected/kube-api-access-pc2b6:{mountpoint:/var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~projected/kube-api-access-pc2b6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/579dfa33-9adb-4613-b055-11671039eea2/volumes/kubernetes.io~projected/kube-api-access-xqc5n:{mountpoint:/var/lib/kubelet/pods/579dfa33-9adb-4613-b055-11671039eea2/volumes/kubernetes.io~projected/kube-api-access-xqc5n major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cfa57d0-764b-42fb-ad36-9ca6dc472766/volumes/kubernetes.io~projected/kube-api-access-thc6l:{mountpoint:/var/lib/kubelet/pods/5cfa57d0-764b-42fb-ad36-9ca6dc472766/volumes/kubernetes.io~projected/kube-api-access-thc6l major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dc70a14-ff5e-4088-ab14-15700a98ad22/volumes/kubernetes.io~projected/kube-api-access-cjtb9:{mountpoint:/var/lib/kubelet/pods/5dc70a14-ff5e-4088-ab14-15700a98ad22/volumes/kubernetes.io~projected/kube-api-access-cjtb9 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60389df0-3bb0-42fd-9a5b-40bfa22bacf5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/60389df0-3bb0-42fd-9a5b-40bfa22bacf5/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5/volumes/kubernetes.io~projected/kube-api-access-jdzdn:{mountpoint:/var/lib/kubelet/pods/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5/volumes/kubernetes.io~projected/kube-api-access-jdzdn major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc/volumes/kubernetes.io~projected/kube-api-access-v9gc8:{mountpoint:/var/lib/kubelet/pods/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc/volumes/kubernetes.io~projected/kube-api-access-v9gc8 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/783ad43d-a72e-4b58-b403-00eab40f6cdc/volumes/kubernetes.io~projected/kube-api-access-vhqnt:{mountpoint:/var/lib/kubelet/pods/783ad43d-a72e-4b58-b403-00eab40f6cdc/volumes/kubernetes.io~projected/kube-api-access-vhqnt major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/783ad43d-a72e-4b58-b403-00eab40f6cdc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/783ad43d-a72e-4b58-b403-00eab40f6cdc/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80541015-4203-4621-bba9-d737aa2f3266/volumes/kubernetes.io~projected/kube-api-access-tcznv:{mountpoint:/var/lib/kubelet/pods/80541015-4203-4621-bba9-d737aa2f3266/volumes/kubernetes.io~projected/kube-api-access-tcznv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80541015-4203-4621-bba9-d737aa2f3266/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/80541015-4203-4621-bba9-d737aa2f3266/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b062870-dd5d-4f35-b400-ea17ebd11878/volumes/kubernetes.io~projected/kube-api-access-t6rnk:{mountpoint:/var/lib/kubelet/pods/8b062870-dd5d-4f35-b400-ea17ebd11878/volumes/kubernetes.io~projected/kube-api-access-t6rnk major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~projected/kube-api-access-ztd6c:{mountpoint:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~projected/kube-api-access-ztd6c major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c4ba53d-6c1a-4032-b83d-225bf88e02e8/volumes/kubernetes.io~projected/kube-api-access-knxln:{mountpoint:/var/lib/kubelet/pods/9c4ba53d-6c1a-4032-b83d-225bf88e02e8/volumes/kubernetes.io~projected/kube-api-access-knxln major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d259950-9b3c-41dc-87bd-74286a0acda0/volumes/kubernetes.io~projected/kube-api-access-fh5fs:{mountpoint:/var/lib/kubelet/pods/9d259950-9b3c-41dc-87bd-74286a0acda0/volumes/kubernetes.io~projected/kube-api-access-fh5fs major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9d98af5-fb06-47ca-a462-8f974bbfd046/volumes/kubernetes.io~projected/kube-api-access-t92wl:{mountpoint:/var/lib/kubelet/pods/a9d98af5-fb06-47ca-a462-8f974bbfd046/volumes/kubernetes.io~projected/kube-api-access-t92wl major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae8165f7-bad3-418a-be14-97acb4604cf7/volumes/kubernetes.io~projected/kube-api-access-rc7x8:{mountpoint:/var/lib/kubelet/pods/ae8165f7-bad3-418a-be14-97acb4604cf7/volumes/kubernetes.io~projected/kube-api-access-rc7x8 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7e4c8da-dd72-438d-bd5b-a81728051a95/volumes/kubernetes.io~projected/kube-api-access-dzfvp:{mountpoint:/var/lib/kubelet/pods/b7e4c8da-dd72-438d-bd5b-a81728051a95/volumes/kubernetes.io~projected/kube-api-access-dzfvp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5b7952d-24a8-40c5-a755-3aeb8269714b/volumes/kubernetes.io~projected/kube-api-access-hh5xs:{mountpoint:/var/lib/kubelet/pods/c5b7952d-24a8-40c5-a755-3aeb8269714b/volumes/kubernetes.io~projected/kube-api-access-hh5xs major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volumes/kubernetes.io~projected/kube-api-access-4hhr6:{mountpoint:/var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volumes/kubernetes.io~projected/kube-api-access-4hhr6 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c79db355-84dd-4f0a-b102-d7c28b1caee6/volumes/kubernetes.io~projected/kube-api-access-vjpc6:{mountpoint:/var/lib/kubelet/pods/c79db355-84dd-4f0a-b102-d7c28b1caee6/volumes/kubernetes.io~projected/kube-api-access-vjpc6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce5465ef-bfd2-4075-8560-a19c26720d7e/volumes/kubernetes.io~projected/kube-api-access-d6tgh:{mountpoint:/var/lib/kubelet/pods/ce5465ef-bfd2-4075-8560-a19c26720d7e/volumes/kubernetes.io~projected/kube-api-access-d6tgh major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b542601c63271b7a8c96620d513505954578263647d0c830fa64b2b07b454a9c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3b21ec9f1f11b1f0305cd27ca7c6df355b8f844e9754499eaf7b6598aaa9d659/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a6e43407a0916bf258fd52b628ded06e102222e2148f6972edc0c5f9b1f2e862/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/f290ef216b323b66447f9df0f87e3842b1de6298fbb2fca59984da290fb3757f/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/80c2f7df84a9c072da922f5b88217fa6e1a89788472ed644acf0901c30d32a77/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6e8f1e112cf698ddd92d05bf913adecb7e6299621a1ff8e10c0c95963518ef3d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e0a3cbcc0fc98c61fe0cc95a50f0115218a09678f4ae32a2e1b746e5076cf29a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2227779a1693ec981ab4e724ed974ee114b04911db23dafb08662e610cab0a40/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7a008458007d1da4cd6c1900e76658420a925ae5aa3ccc617623fc5f4fe3c815/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9d4df425320b5edcf4082fb6060f24b6883780dc73fffccec9a1426c4884d02f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2e24c6de03c52fbb2d46a25f9c8c794276909476aa3d8f261a320dfd362a8ecb/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/d3a340e9a4a9696fbf28d053e26ba9342c2b2d3d4e6059c477339a91edb236c9/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/cda8d87154347c3a8d42d738ef9a5da7a81d75d507ecaedc2334f3515fdaabdd/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/faa17166a4196fbe23d360fcf65bb4844a9f562ef9d03d392d19c7f68ce50499/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0975b7509e843e21a89674c7e27513b10a1565af1ce5a4107704e177650b00e9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d5b2bee32f0f95d82cddb186d484ff79d842f9b846a2bb318c005fab94ccac97/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e91edd2e9c0fd06fb5644da6ac51a9f1f3fb8db7ae6c5932dd412511b4a5922e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/30bc6e8b58678c5afae3f06f801796445696c72906d0585658bb51d640fb703d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/95e7731601848d33930f1ecaded6b1eddeb6b7134a4094f2afa490f0cca2b076/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a2309f8e020988b998a211349c96423b9d7d61a127f7d57044b495df7d1ad123/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b9f9152bb318a3e6739383ac878e918bc03d7fb905baace70cd2dc4dcc91c7df/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/99aef5404bab7bb9c4d9b5f5787e430a898b860b623a126496b64cd4458a8fdb/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8f494b506e64359006c4d318fbe355dfd5cf66a0aedc69f3b1fcf2cb5803f296/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/cd8cc0a2160e5a56574cf1d75ce61bf4c80d20a4a100554aa618dd17281327be/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/56352da961d012e108c66819b4845ee5c0a7fb09d87300b6219ff9c578a39ebd/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/e8f11caba68fee26dd23998318c353ebf6e97e7d36999e5415c4ea9bff33681f/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/9f304f9c3ecbca6feecd7aaaf05cf8cf2a89a8ba50c0a1b5333a8dba48ef3bec/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/bc82571b0a741db949ab1d10ec4d8c1d4897dd8ed99364b49a31e74377ba08b3/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/31b4ab1e3749b67f463aabe9d5104b536aa1173b4d972cd46986901504f82e76/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/cc5c753d55b9efa004f4544cee463fb37caae54a49cd7c9c74d2ebd0dd4d07ed/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/40b7e12b7a8275e14f30062411a428cb94925d1ca906a4fd72f19fcdfdd4b928/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/ca192fb2fe93bd1b92d3487f501adfbfb0236eca158a19b418a0309916fd5243/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/68229372c8019a328ebabefffbea53ab38c175b565b39e429b4280bdd176afb1/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/4a1c4b7d8fd10280823605b91e43d5217b48765167c41ebee9147700bdd32489/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/ec2ba1bcf81da1a8c5b400d4601663ef5de6a09ce182aff960b96674a241a513/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/72a6f039e6154e1481bf1441a29b058704381007d0ffc887cdf4e63a902b3292/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/3ae3ca6eea175af1f9804dde2a5e723e25b362a04c59271568e7bd1331a60edf/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/ea4aa6711f95d3d7435b1442e4daf91be41660bc8d33671be33e4d68a6f13f19/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f7d0d671854c7fbdd04e547fd5dbb56ae92669ab87b7a5501cf672b1287cad6c/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/7a611a33f0dde02f0727bfadfaa5efd8e39adb11bc53d5331c68b9cd2fb82ae9/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/7319284518e15845fbddd1f0cf70b5f89a32b9dec897bbbfe0539f386662bfdd/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/aaf2ead6c1be44211a1cf01877f0e0336a146eb0cd721ada735e900627b41292/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/a0c1cef7161b4cd589ca26aec4218f9e9a566ec53c451e1b7503b80649513099/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/2194eefbf4773fe21ce8a5239caca6046f658148e87264fefaec45044e76636d/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/cae5c69b7dce3291c6a029b949d913edae13ad1371f04196deb2aca6922f05ad/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fbb794a2c357274912b942fb3f87b27a1475f5d1b7eb6ddd26e19b10f75caa42/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2d83b8c2b57e1de81ae60d1fae15d7beb6b605da456b48d31bd9d83453a46962/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/926535fa2682e6a5075ef65a0299b5ad09446b5762532da60f060dcc70f762e1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c43f36e2eca0f7388c04ff882b12970f0aa99f8036d5146320565e744894201f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/966d2b6ab37193c0f62fd307134b6cb8b668d66699a44da5bc81f58e0f387f52/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ce329ac1a9eb920487787fb2e2f100136cf0f1b132865b45c14acf4babcefc02/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d2299b40b474dbc0218d307f89c48736bb584914c4b56d46ef1443ec06fbd3f4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e5bc8a4585443c2c6180555229788a7c94d612495004077e11d7515375bbd2df/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c07217c4cad9cf3c003dd7455b1ef5888e0e31fa5d5a55d936d646f1109653fc/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/32b9e891ef09a8806736b6c2e166fe6e2d2d2c54bcc2f265c6c48a0ce8fe3f33/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.117088 4910 manager.go:217] Machine: {Timestamp:2026-11-01 00:23:30.116200282 +0000 UTC m=+0.095782636 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101010 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22d0086a4d0937f0c3429601604d8e SystemUUID:ec22d008-6a4d-0937-f0c3-429601604d8e BootID:f3398203-3447-4e69-b843-a1624bbe2a8e Filesystems:[{Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/252a2753-2639-4920-afaf-0fb0ca316d26/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce5465ef-bfd2-4075-8560-a19c26720d7e/volumes/kubernetes.io~projected/kube-api-access-d6tgh DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80541015-4203-4621-bba9-d737aa2f3266/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/783ad43d-a72e-4b58-b403-00eab40f6cdc/volumes/kubernetes.io~projected/kube-api-access-vhqnt DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79d68294b96ef2da5fc23df895defd47c35e4ca6f82bc11a21e21cf65baddb4b/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ac18260-596d-49f0-b109-d5750e8c3644/volumes/kubernetes.io~projected/kube-api-access-8k7vb DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f99bf252830c0acab95b3f4507930a31d6bbf9858daa9264855613fa09492337/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf6725911278feba1375869271551c31483a5aa62d14a4360cb97b8db91fc654/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/60389df0-3bb0-42fd-9a5b-40bfa22bacf5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4454de50-086b-4842-8e00-d2ffae8344d2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a052aef8177eb60c0002f77bf693a10077a9e50abe7de0f21054ba97cd8fd68/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b734bc2834027f0b0318f25842ed43adebcb55f86afe9034a00589c6ae01a414/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc/volumes/kubernetes.io~projected/kube-api-access-v9gc8 DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3246fb55-62ae-4d36-a951-75206d4f99d1/volumes/kubernetes.io~projected/kube-api-access-5twnl DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3b775de6-474a-4551-8ddc-ade7a8c25fab/volumes/kubernetes.io~projected/kube-api-access-hsgbt DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/348d2cd331244881640bfba75d36fde3ce1d8dd8e4961e785deb568be8641486/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d70a785fe5c03e3777f250930ba2f8965e203e831897847e72e757ec855f9788/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5b7952d-24a8-40c5-a755-3aeb8269714b/volumes/kubernetes.io~projected/kube-api-access-hh5xs DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cfa57d0-764b-42fb-ad36-9ca6dc472766/volumes/kubernetes.io~projected/kube-api-access-thc6l DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80fda66eb807be3c9b964db1d67e0ad1ecbeb3ce6d6648b82c2567d4909e5d05/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7e4c8da-dd72-438d-bd5b-a81728051a95/volumes/kubernetes.io~projected/kube-api-access-dzfvp DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~projected/kube-api-access-ztd6c DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c547bac0fa98b6c59c24fa02eedcd524090e44af2fb6a3bc43bc18e3d818b146/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c79db355-84dd-4f0a-b102-d7c28b1caee6/volumes/kubernetes.io~projected/kube-api-access-vjpc6 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volumes/kubernetes.io~projected/kube-api-access-4hhr6 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2375a2b5cd32d8cd8837b3ac14dc0910d62d65c1d80caaf0ef32c22accefaa7/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38761a0d-2c71-4f3b-8d5d-39cdf321763f/volumes/kubernetes.io~projected/kube-api-access-2j6l2 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/252a2753-2639-4920-afaf-0fb0ca316d26/volumes/kubernetes.io~projected/kube-api-access-wghrw DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~projected/kube-api-access-lsw6b DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80541015-4203-4621-bba9-d737aa2f3266/volumes/kubernetes.io~projected/kube-api-access-tcznv DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ae8165f7-bad3-418a-be14-97acb4604cf7/volumes/kubernetes.io~projected/kube-api-access-rc7x8 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f765550379f004449a7a7c7498e53f43ee64939579ef6aeca79bd25b17f5aeb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17b3477894b9d5e00f72a358811919763a52c4a9d342b547aad4939a9ffee860/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efcfa88b3b73200463e02d198abe5e0182425c7e44b154aaaca2c257f8ab8f23/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05423de021baa9deb6552031a4ebf17180a97b8365774aada80623b416242df3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4dddafc0d945618c75fa2248ac9b8cad882926f88782338b01e9bdef75bedab/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ac18260-596d-49f0-b109-d5750e8c3644/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2eebd2c83e9f1b40e78d7a69756eff17c19860ed63ccbf708465559ec91f82d5/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/783ad43d-a72e-4b58-b403-00eab40f6cdc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c4ba53d-6c1a-4032-b83d-225bf88e02e8/volumes/kubernetes.io~projected/kube-api-access-knxln DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/577842867deee98b9a5c8ae987761f7eec82d085bca8668087abca837c5a184c/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fca98fa50f03b97afd456d5cc8bab5898efb5e4b3bf5bfb529b7928b124efe12/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d259950-9b3c-41dc-87bd-74286a0acda0/volumes/kubernetes.io~projected/kube-api-access-fh5fs DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e5dc3f7f009d302e4ed224a24f80844223df0caf8ee4d91ec321a472b10bf33/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~projected/kube-api-access-pc2b6 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5dc70a14-ff5e-4088-ab14-15700a98ad22/volumes/kubernetes.io~projected/kube-api-access-cjtb9 DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a9d98af5-fb06-47ca-a462-8f974bbfd046/volumes/kubernetes.io~projected/kube-api-access-t92wl DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df5599d793d24509604a61bebdc994ce74c1c9d926a7a3da161dfed6d4ac326f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~projected/kube-api-access-dwpgz DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7/volumes/kubernetes.io~projected/kube-api-access-dh8m6 DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2508f7ff-f9da-48fe-8e1c-ba02568e0899/volumes/kubernetes.io~projected/kube-api-access-nngm4 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b062870-dd5d-4f35-b400-ea17ebd11878/volumes/kubernetes.io~projected/kube-api-access-t6rnk DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/579dfa33-9adb-4613-b055-11671039eea2/volumes/kubernetes.io~projected/kube-api-access-xqc5n DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c767a0e3-268a-423f-8113-f98ab65fa4b3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d85dc68-06e8-4036-93b1-ececa1a779c7/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5/volumes/kubernetes.io~projected/kube-api-access-jdzdn DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/075828c4fa1dc0f9dfbaac5ebccdd8ca9a8c7198f5f75687ba2c3e4c215a4c68/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/48f9765c-72a2-4a67-b161-9de71454b8cc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:075828c4fa1dc0f MacAddress:0a:6b:dd:d7:bc:85 Speed:10000 Mtu:8901} {Name:17b3477894b9d5e MacAddress:46:76:d5:1b:cd:7e Speed:10000 Mtu:8901} {Name:2eebd2c83e9f1b4 MacAddress:4e:9a:0e:39:5c:f8 Speed:10000 Mtu:8901} {Name:348d2cd33124488 MacAddress:f6:ec:da:8d:d0:24 Speed:10000 Mtu:8901} {Name:577842867deee98 MacAddress:3a:81:38:b3:0b:76 Speed:10000 Mtu:8901} {Name:79d68294b96ef2d MacAddress:42:18:f2:6c:a5:2e Speed:10000 Mtu:8901} {Name:9e5dc3f7f009d30 MacAddress:16:96:d1:d4:7b:cc Speed:10000 Mtu:8901} {Name:b4dddafc0d94561 MacAddress:ea:ed:7a:7c:1b:ef Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:55:fa:0c:3f:c1 Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:b0:43:7a:28:53 Speed:0 Mtu:8901} {Name:c547bac0fa98b6c MacAddress:be:49:d3:e6:20:06 Speed:10000 Mtu:8901} {Name:cf6725911278feb MacAddress:66:56:1c:07:78:33 Speed:10000 Mtu:8901} {Name:d70a785fe5c03e3 MacAddress:56:9a:e0:63:0d:ec Speed:10000 Mtu:8901} {Name:efcfa88b3b73200 MacAddress:8e:26:11:70:5d:10 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:55:fa:0c:3f:c1 Speed:0 Mtu:9001} {Name:fca98fa50f03b97 MacAddress:da:c9:cd:f1:64:39 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:86:9d:fe:48:72:98 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3a:11:63:1a:05:a1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.117652 4910 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.117715 4910 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118082 4910 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118107 4910 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-150.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118291 4910 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118299 4910 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118310 4910 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118332 4910 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118359 4910 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118560 4910 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118714 4910 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118753 4910 kubelet.go:493] "Attempting to sync node with API server" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118768 4910 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118781 4910 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118793 4910 kubelet.go:405] "Adding apiserver pod source" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.118805 4910 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.119673 4910 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.119686 4910 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:23:30.121198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.120268 4910 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 00:23:30.138311 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138290 4910 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 00:23:30.138311 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138309 4910 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 00:23:30.138488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138426 4910 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 00:23:30.138684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138661 4910 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 00:23:30.138744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138702 4910 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 00:23:30.138847 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138837 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 00:23:30.138879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138852 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 00:23:30.138879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138859 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 00:23:30.138879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138865 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 00:23:30.138879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138875 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138885 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138892 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138897 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138907 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138918 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138930 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138940 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138968 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 00:23:30.139016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.138974 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 00:23:30.142988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.142973 4910 server.go:1263] "Started kubelet" Nov 01 00:23:30.143246 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.143189 4910 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 00:23:30.143317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.143268 4910 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 00:23:30.143414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.143371 4910 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 00:23:30.143533 ip-10-0-137-150 systemd[1]: Started Kubernetes Kubelet. Nov 01 00:23:30.143669 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.143545 4910 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 00:23:30.144738 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.144722 4910 server.go:317] "Adding debug handlers to kubelet server" Nov 01 00:23:30.146455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.146435 4910 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.146683 4910 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.146717 4910 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.146757 4910 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.146766 4910 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.146836 4910 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.147001 4910 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 00:17:50 +0000 UTC" deadline="2028-06-01 17:53:37.150910535 +0000 UTC" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.147021 4910 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13889h30m7.003892252s" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149737 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e97e00-f916-46ea-bcb1-2f0f52ebe2e5" volumeName="kubernetes.io/secret/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-cert" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149773 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c767a0e3-268a-423f-8113-f98ab65fa4b3" volumeName="kubernetes.io/projected/c767a0e3-268a-423f-8113-f98ab65fa4b3-kube-api-access-4hhr6" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149783 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f9765c-72a2-4a67-b161-9de71454b8cc" volumeName="kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-tmp" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149792 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f9765c-72a2-4a67-b161-9de71454b8cc" volumeName="kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-tuned" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149808 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2508f7ff-f9da-48fe-8e1c-ba02568e0899" volumeName="kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149815 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="783ad43d-a72e-4b58-b403-00eab40f6cdc" volumeName="kubernetes.io/configmap/783ad43d-a72e-4b58-b403-00eab40f6cdc-config" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149823 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98642830-5eb4-42bf-8c4a-7ffabdc5d965" volumeName="kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149831 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9d98af5-fb06-47ca-a462-8f974bbfd046" volumeName="kubernetes.io/projected/a9d98af5-fb06-47ca-a462-8f974bbfd046-kube-api-access-t92wl" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149840 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5b7952d-24a8-40c5-a755-3aeb8269714b" volumeName="kubernetes.io/projected/c5b7952d-24a8-40c5-a755-3aeb8269714b-kube-api-access-hh5xs" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149848 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d85dc68-06e8-4036-93b1-ececa1a779c7" volumeName="kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-stats-auth" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149859 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80541015-4203-4621-bba9-d737aa2f3266" volumeName="kubernetes.io/secret/80541015-4203-4621-bba9-d737aa2f3266-serving-cert" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149870 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9d98af5-fb06-47ca-a462-8f974bbfd046" volumeName="kubernetes.io/empty-dir/a9d98af5-fb06-47ca-a462-8f974bbfd046-tmp-dir" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149882 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae8165f7-bad3-418a-be14-97acb4604cf7" volumeName="kubernetes.io/configmap/ae8165f7-bad3-418a-be14-97acb4604cf7-iptables-alerter-script" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149892 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0316e070-2f84-4652-82ce-51837acdc06e" volumeName="kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149902 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ac18260-596d-49f0-b109-d5750e8c3644" volumeName="kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-service-ca-bundle" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149909 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ac18260-596d-49f0-b109-d5750e8c3644" volumeName="kubernetes.io/projected/0ac18260-596d-49f0-b109-d5750e8c3644-kube-api-access-8k7vb" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149917 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a2753-2639-4920-afaf-0fb0ca316d26" volumeName="kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-trusted-ca" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149925 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4454de50-086b-4842-8e00-d2ffae8344d2" volumeName="kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149932 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d259950-9b3c-41dc-87bd-74286a0acda0" volumeName="kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-daemon-config" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149940 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0316e070-2f84-4652-82ce-51837acdc06e" volumeName="kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149949 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38761a0d-2c71-4f3b-8d5d-39cdf321763f" volumeName="kubernetes.io/projected/38761a0d-2c71-4f3b-8d5d-39cdf321763f-kube-api-access-2j6l2" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149956 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dc70a14-ff5e-4088-ab14-15700a98ad22" volumeName="kubernetes.io/configmap/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-config" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149964 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc" volumeName="kubernetes.io/projected/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc-kube-api-access-v9gc8" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149971 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c79db355-84dd-4f0a-b102-d7c28b1caee6" volumeName="kubernetes.io/projected/c79db355-84dd-4f0a-b102-d7c28b1caee6-kube-api-access-vjpc6" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149979 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="783ad43d-a72e-4b58-b403-00eab40f6cdc" volumeName="kubernetes.io/projected/783ad43d-a72e-4b58-b403-00eab40f6cdc-kube-api-access-vhqnt" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149986 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4ba53d-6c1a-4032-b83d-225bf88e02e8" volumeName="kubernetes.io/projected/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-kube-api-access-knxln" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.149993 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c767a0e3-268a-423f-8113-f98ab65fa4b3" volumeName="kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-env-overrides" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150000 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c767a0e3-268a-423f-8113-f98ab65fa4b3" volumeName="kubernetes.io/secret/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150010 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e97e00-f916-46ea-bcb1-2f0f52ebe2e5" volumeName="kubernetes.io/projected/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-kube-api-access-jdzdn" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150020 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80541015-4203-4621-bba9-d737aa2f3266" volumeName="kubernetes.io/projected/80541015-4203-4621-bba9-d737aa2f3266-kube-api-access-tcznv" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150027 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b062870-dd5d-4f35-b400-ea17ebd11878" volumeName="kubernetes.io/projected/8b062870-dd5d-4f35-b400-ea17ebd11878-kube-api-access-t6rnk" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150035 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98642830-5eb4-42bf-8c4a-7ffabdc5d965" volumeName="kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150042 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60389df0-3bb0-42fd-9a5b-40bfa22bacf5" volumeName="kubernetes.io/secret/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-agent-certs" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150056 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98642830-5eb4-42bf-8c4a-7ffabdc5d965" volumeName="kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150063 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98642830-5eb4-42bf-8c4a-7ffabdc5d965" volumeName="kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150070 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce5465ef-bfd2-4075-8560-a19c26720d7e" volumeName="kubernetes.io/empty-dir/ce5465ef-bfd2-4075-8560-a19c26720d7e-tmp-dir" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150077 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="579dfa33-9adb-4613-b055-11671039eea2" volumeName="kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150085 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dc70a14-ff5e-4088-ab14-15700a98ad22" volumeName="kubernetes.io/projected/5dc70a14-ff5e-4088-ab14-15700a98ad22-kube-api-access-cjtb9" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150092 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9d98af5-fb06-47ca-a462-8f974bbfd046" volumeName="kubernetes.io/configmap/a9d98af5-fb06-47ca-a462-8f974bbfd046-config-volume" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150100 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4cab1f4-47e1-457c-a182-692b3be9bd16" volumeName="kubernetes.io/configmap/b4cab1f4-47e1-457c-a182-692b3be9bd16-nginx-conf" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150107 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0316e070-2f84-4652-82ce-51837acdc06e" volumeName="kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150115 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0316e070-2f84-4652-82ce-51837acdc06e" volumeName="kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150126 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d85dc68-06e8-4036-93b1-ececa1a779c7" volumeName="kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-default-certificate" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150133 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2508f7ff-f9da-48fe-8e1c-ba02568e0899" volumeName="kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150140 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2508f7ff-f9da-48fe-8e1c-ba02568e0899" volumeName="kubernetes.io/projected/2508f7ff-f9da-48fe-8e1c-ba02568e0899-kube-api-access-nngm4" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150148 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b775de6-474a-4551-8ddc-ade7a8c25fab" volumeName="kubernetes.io/projected/3b775de6-474a-4551-8ddc-ade7a8c25fab-kube-api-access-hsgbt" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150155 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4ba53d-6c1a-4032-b83d-225bf88e02e8" volumeName="kubernetes.io/configmap/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-serviceca" seLinuxMountContext="" Nov 01 00:23:30.150202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150162 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae8165f7-bad3-418a-be14-97acb4604cf7" volumeName="kubernetes.io/projected/ae8165f7-bad3-418a-be14-97acb4604cf7-kube-api-access-rc7x8" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150169 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3246fb55-62ae-4d36-a951-75206d4f99d1" volumeName="kubernetes.io/empty-dir/3246fb55-62ae-4d36-a951-75206d4f99d1-data-volume" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150422 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="783ad43d-a72e-4b58-b403-00eab40f6cdc" volumeName="kubernetes.io/secret/783ad43d-a72e-4b58-b403-00eab40f6cdc-serving-cert" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150449 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80541015-4203-4621-bba9-d737aa2f3266" volumeName="kubernetes.io/configmap/80541015-4203-4621-bba9-d737aa2f3266-config" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150472 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98642830-5eb4-42bf-8c4a-7ffabdc5d965" volumeName="kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150486 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7e4c8da-dd72-438d-bd5b-a81728051a95" volumeName="kubernetes.io/projected/b7e4c8da-dd72-438d-bd5b-a81728051a95-kube-api-access-dzfvp" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150506 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ac18260-596d-49f0-b109-d5750e8c3644" volumeName="kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-trusted-ca-bundle" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150521 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ac18260-596d-49f0-b109-d5750e8c3644" volumeName="kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-tmp" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150535 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ac18260-596d-49f0-b109-d5750e8c3644" volumeName="kubernetes.io/secret/0ac18260-596d-49f0-b109-d5750e8c3644-serving-cert" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150578 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ba5db8b-e910-4ba0-b604-d1a8b393d1f7" volumeName="kubernetes.io/configmap/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-telemetry-config" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150591 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a2753-2639-4920-afaf-0fb0ca316d26" volumeName="kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-config" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150603 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cfa57d0-764b-42fb-ad36-9ca6dc472766" volumeName="kubernetes.io/projected/5cfa57d0-764b-42fb-ad36-9ca6dc472766-kube-api-access-thc6l" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150736 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60389df0-3bb0-42fd-9a5b-40bfa22bacf5" volumeName="kubernetes.io/configmap/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-konnectivity-ca" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150756 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d259950-9b3c-41dc-87bd-74286a0acda0" volumeName="kubernetes.io/projected/9d259950-9b3c-41dc-87bd-74286a0acda0-kube-api-access-fh5fs" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150776 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0316e070-2f84-4652-82ce-51837acdc06e" volumeName="kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150802 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0316e070-2f84-4652-82ce-51837acdc06e" volumeName="kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150814 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98642830-5eb4-42bf-8c4a-7ffabdc5d965" volumeName="kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150832 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce5465ef-bfd2-4075-8560-a19c26720d7e" volumeName="kubernetes.io/projected/ce5465ef-bfd2-4075-8560-a19c26720d7e-kube-api-access-d6tgh" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150846 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ac18260-596d-49f0-b109-d5750e8c3644" volumeName="kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-snapshots" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150865 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d85dc68-06e8-4036-93b1-ececa1a779c7" volumeName="kubernetes.io/projected/0d85dc68-06e8-4036-93b1-ececa1a779c7-kube-api-access-lsw6b" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150881 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2508f7ff-f9da-48fe-8e1c-ba02568e0899" volumeName="kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-binary-copy" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150893 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98642830-5eb4-42bf-8c4a-7ffabdc5d965" volumeName="kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150910 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d259950-9b3c-41dc-87bd-74286a0acda0" volumeName="kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-cni-binary-copy" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150937 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c767a0e3-268a-423f-8113-f98ab65fa4b3" volumeName="kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-config" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150956 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ba5db8b-e910-4ba0-b604-d1a8b393d1f7" volumeName="kubernetes.io/projected/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-kube-api-access-dh8m6" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150969 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a2753-2639-4920-afaf-0fb0ca316d26" volumeName="kubernetes.io/projected/252a2753-2639-4920-afaf-0fb0ca316d26-kube-api-access-wghrw" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.150993 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c767a0e3-268a-423f-8113-f98ab65fa4b3" volumeName="kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-script-lib" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.151012 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0316e070-2f84-4652-82ce-51837acdc06e" volumeName="kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.151024 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252a2753-2639-4920-afaf-0fb0ca316d26" volumeName="kubernetes.io/secret/252a2753-2639-4920-afaf-0fb0ca316d26-serving-cert" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.151040 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3246fb55-62ae-4d36-a951-75206d4f99d1" volumeName="kubernetes.io/projected/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-api-access-5twnl" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.151050 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f9765c-72a2-4a67-b161-9de71454b8cc" volumeName="kubernetes.io/projected/48f9765c-72a2-4a67-b161-9de71454b8cc-kube-api-access-pc2b6" seLinuxMountContext="" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.151061 4910 reconstruct.go:97] "Volume reconstruction finished" Nov 01 00:23:30.152700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.151068 4910 reconciler.go:29] "Reconciler: start to sync state" Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153104 4910 factory.go:153] Registering CRI-O factory Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153119 4910 factory.go:223] Registration of the crio container factory successfully Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153170 4910 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153180 4910 factory.go:55] Registering systemd factory Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153186 4910 factory.go:223] Registration of the systemd container factory successfully Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153206 4910 factory.go:103] Registering Raw factory Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153221 4910 manager.go:1199] Started watching for new ooms in manager Nov 01 00:23:30.153884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.153500 4910 manager.go:319] Starting recovery of all containers Nov 01 00:23:30.170582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.170556 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/1.log" Nov 01 00:23:30.172896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.172872 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/1.log" Nov 01 00:23:30.181710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.181674 4910 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 00:23:30.183692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.183667 4910 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 00:23:30.183692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.183696 4910 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 00:23:30.183852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.183735 4910 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 00:23:30.183905 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.183786 4910 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 00:23:30.209020 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.208987 4910 generic.go:356] "Generic (PLEG): container finished" podID="1c3f780546aa667fa9a0746baec62e3e" containerID="329c71348bce591be625869ef3f40c459d9776c13a09868a7505a14d30155208" exitCode=0 Nov 01 00:23:30.224852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.224797 4910 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="42be5f0e621e51fb92f2c452e89fd765262d03f2142af635a9f08ae3ab094a8d" exitCode=0 Nov 01 00:23:30.224852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.224835 4910 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="32d83023b7fda1bb227cf52f5a50e5b25ec8ce53a32c0469c4ea657be1fe46b2" exitCode=0 Nov 01 00:23:30.224852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.224846 4910 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="cf4be57ac545d98e3027f56468c5dbb4017a90f793840f013334348dec6ee9b0" exitCode=0 Nov 01 00:23:30.224852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.224855 4910 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="4be3322458df8ec48fd1cfd154c336dd4d26ee29448854246db24753451f1bb8" exitCode=0 Nov 01 00:23:30.225104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.224864 4910 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="17eabcbf35d474a9b69d9afd5af6e5165a9775e8b93078037cfa732b8eb29ddd" exitCode=0 Nov 01 00:23:30.225104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.224873 4910 generic.go:356] "Generic (PLEG): container finished" podID="2508f7ff-f9da-48fe-8e1c-ba02568e0899" containerID="c4d6c3ed301ec01dbbac0c088abc486b7b3e513a745e1387a0057444cd2852ed" exitCode=0 Nov 01 00:23:30.246803 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.246773 4910 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:23:30.257009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.256988 4910 manager.go:324] Recovery completed Nov 01 00:23:30.260830 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.260800 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/1.log" Nov 01 00:23:30.260962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.260854 4910 generic.go:356] "Generic (PLEG): container finished" podID="252a2753-2639-4920-afaf-0fb0ca316d26" containerID="1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd" exitCode=255 Nov 01 00:23:30.274994 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.274974 4910 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:30.276076 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.276059 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:30.276161 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.276094 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:30.276161 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.276108 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:30.278292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278272 4910 cpu_manager.go:226] "Starting" policy="none" Nov 01 00:23:30.278292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278285 4910 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:30.278292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278288 4910 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 00:23:30.278466 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278308 4910 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 00:23:30.278732 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278712 4910 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 00:23:30.278828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278732 4910 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 00:23:30.278828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278758 4910 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 00:23:30.278828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278766 4910 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 00:23:30.278828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278774 4910 policy_none.go:50] "Start" Nov 01 00:23:30.278828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278797 4910 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 00:23:30.278828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.278810 4910 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 00:23:30.279200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.279180 4910 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 00:23:30.279250 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.279202 4910 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 00:23:30.279250 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.279212 4910 policy_none.go:44] "Start" Nov 01 00:23:30.281266 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.281242 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:30.281569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.281552 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:30.281617 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.281591 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:30.284250 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.284222 4910 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 00:23:30.285465 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.285432 4910 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 00:23:30.290281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.290268 4910 manager.go:341] "Starting Device Plugin manager" Nov 01 00:23:30.290358 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.290292 4910 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 00:23:30.290358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.290301 4910 server.go:87] "Starting device plugin registration server" Nov 01 00:23:30.290599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.290586 4910 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 00:23:30.291092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.290603 4910 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 00:23:30.291179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.291132 4910 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 00:23:30.292014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.291977 4910 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 00:23:30.292014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.292021 4910 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 00:23:30.292014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.292028 4910 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 00:23:30.292207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.292090 4910 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T00:23:30.292010306Z","UUID":"9338e0a9-4f21-4efb-9c92-05122d860caa","Handler":null,"Name":"","Endpoint":""} Nov 01 00:23:30.292307 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.292293 4910 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 00:23:30.292349 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.292326 4910 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-150.us-west-1.compute.internal\" not found" Nov 01 00:23:30.293436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.293417 4910 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 00:23:30.293537 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.293441 4910 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 00:23:30.330141 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.330115 4910 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:30.332278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.332253 4910 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:30.332394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.332303 4910 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:30.332394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.332381 4910 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 00:23:30.382971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.382946 4910 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 00:23:30.391922 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.391909 4910 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 00:23:30.392952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.392936 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 00:23:30.393003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.392965 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 00:23:30.393003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.392977 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-150.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 00:23:30.393093 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.393017 4910 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.484400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.484284 4910 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal"] Nov 01 00:23:30.485012 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.484978 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"1c3f780546aa667fa9a0746baec62e3e","Type":"ContainerStarted","Data":"e7ac07b1a194f408b201a44cd8faacd59c86dc6edb9d48ba2fb7d6e8cbe56bba"} Nov 01 00:23:30.485060 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.485020 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"1c3f780546aa667fa9a0746baec62e3e","Type":"ContainerDied","Data":"329c71348bce591be625869ef3f40c459d9776c13a09868a7505a14d30155208"} Nov 01 00:23:30.485060 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.485036 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"1c3f780546aa667fa9a0746baec62e3e","Type":"ContainerStarted","Data":"0e4d2755fadb1e27c4243fdc5fdfa04907164581ace4631834a99cd8279a0760"} Nov 01 00:23:30.485136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.485105 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"f0c4975681f29214750bf74479eb4135","Type":"ContainerStarted","Data":"667bc430c0b2d74090dd86db69f9aa52b152336759dcd6f7c4439e40ff10edc2"} Nov 01 00:23:30.485136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.485118 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" event={"ID":"f0c4975681f29214750bf74479eb4135","Type":"ContainerStarted","Data":"a0a36df7c2c8cf6fb998219870f80c45da47c343276dad28f0e21a17689bf185"} Nov 01 00:23:30.485208 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.485150 4910 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.485238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.485208 4910 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.502797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.502768 4910 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.502926 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.502842 4910 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.546634 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.546606 4910 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:23:30.546790 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.546677 4910 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.547431 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.547413 4910 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:23:30.547472 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:30.547458 4910 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.633843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.633815 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.634013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.633851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0c4975681f29214750bf74479eb4135-config\") pod \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"f0c4975681f29214750bf74479eb4135\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.634013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.633932 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.734511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.734434 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.734511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.734388 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.734511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.734499 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.734734 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.734551 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3f780546aa667fa9a0746baec62e3e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"1c3f780546aa667fa9a0746baec62e3e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.734734 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.734578 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0c4975681f29214750bf74479eb4135-config\") pod \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"f0c4975681f29214750bf74479eb4135\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:30.734734 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:30.734615 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0c4975681f29214750bf74479eb4135-config\") pod \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" (UID: \"f0c4975681f29214750bf74479eb4135\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:31.119410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.119377 4910 apiserver.go:52] "Watching apiserver" Nov 01 00:23:31.264350 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.264329 4910 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:31.264501 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.264477 4910 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:31.273436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.273418 4910 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 00:23:31.274092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.274072 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm","openshift-console-operator/console-operator-7cd56b8dbc-nq7f9","openshift-dns/dns-default-bvzrd","kube-system/konnectivity-agent-vtb29","openshift-ingress-canary/ingress-canary-5hhd5","openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz","openshift-network-operator/iptables-alerter-gwksj","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4","openshift-ingress/router-default-69cbfc858f-rq59s","kube-system/kas-connection-checker-685895c7c9-fxsjd","openshift-network-console/networking-console-plugin-86c478bb88-7vncn","openshift-image-registry/image-registry-649fb6d9c5-llkn8","openshift-multus/multus-additional-cni-plugins-6qv8g","openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5","openshift-insights/insights-runtime-extractor-k2s97","openshift-image-registry/node-ca-kl4cn","openshift-insights/insights-operator-7c4986d449-srh5n","openshift-multus/network-metrics-daemon-dtpjl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v","openshift-image-registry/image-registry-78f6fdc666-mgj5x","openshift-dns/node-resolver-v47fq","openshift-network-diagnostics/network-check-target-bq86m","openshift-cluster-node-tuning-operator/tuned-g77tg","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds","kube-system/kas-connection-checker-685895c7c9-gsr76","openshift-ovn-kubernetes/ovnkube-node-wg9vt","kube-system/global-pull-secret-syncer-m45bc","openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb","openshift-multus/multus-f66hm","kube-system/kas-connection-checker-685895c7c9-7btpl","kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal"] Nov 01 00:23:31.275438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.275420 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:31.276962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.276941 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.277708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.277689 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:31.278085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.278068 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:31.278414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.278400 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.283615 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.283589 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:31.287425 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.287406 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.287903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.287890 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.288937 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.288917 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.325299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.325277 4910 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:23:31.325419 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.325326 4910 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:31.325862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.325848 4910 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 00:23:31.325897 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.325885 4910 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-150.us-west-1.compute.internal" Nov 01 00:23:31.327611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.327594 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 00:23:31.327686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.327600 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-z4ffh\"" Nov 01 00:23:31.327686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.327648 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lfxl8\"" Nov 01 00:23:31.328165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.328150 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.328214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.328170 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.328254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.328209 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 00:23:31.328506 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.328490 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2pkrq\"" Nov 01 00:23:31.328675 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.328661 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gb7fs\"" Nov 01 00:23:31.328915 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.328902 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.328997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.328986 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 00:23:31.329153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.329125 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.329302 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.329289 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.329754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.329732 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 00:23:31.329877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.329863 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.329935 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.329847 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 00:23:31.329983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.329975 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.330165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330152 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.330165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330161 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.330257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330208 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.330257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330214 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 00:23:31.330257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330226 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 00:23:31.330351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330257 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.330351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330288 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.330417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330358 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.330455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330445 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 00:23:31.330541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.330529 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.331109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.331095 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cvjqf\"" Nov 01 00:23:31.331371 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.331354 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 00:23:31.331409 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.331369 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 00:23:31.331642 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.331617 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ntf5k\"" Nov 01 00:23:31.349608 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.349587 4910 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 00:23:31.427691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427587 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.427691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427648 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.427865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427713 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.427929 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427915 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.427979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427935 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.427979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427935 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.427979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427959 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.428108 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.427980 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.428108 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428006 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 00:23:31.428108 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428041 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 00:23:31.428108 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428069 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.428108 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428105 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.428310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428130 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 00:23:31.428310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428189 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 00:23:31.428310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428216 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.428310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428224 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 00:23:31.428310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428240 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.428511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428375 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.428767 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428745 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.428905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428788 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 00:23:31.428905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.428795 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.429103 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429084 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 00:23:31.429103 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429099 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 00:23:31.429288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429134 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 00:23:31.429288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429141 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 00:23:31.429288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429134 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.429288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429180 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 00:23:31.429448 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429343 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nz8nc\"" Nov 01 00:23:31.429448 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429364 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.429448 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429377 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.429552 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429454 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.429801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429777 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.429801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429794 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 00:23:31.429901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429880 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 00:23:31.430008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429900 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 00:23:31.430008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429906 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 00:23:31.430008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429941 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 00:23:31.430008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429967 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 00:23:31.430008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429977 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.430008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.429917 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 00:23:31.430205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430148 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2d2tf\"" Nov 01 00:23:31.430205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430159 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 00:23:31.430205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430202 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 00:23:31.430295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430235 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 00:23:31.430332 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430310 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 00:23:31.430432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430409 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 00:23:31.430507 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430434 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.430507 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430441 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 00:23:31.430507 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430454 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 00:23:31.430825 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430801 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhlh7\"" Nov 01 00:23:31.430974 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.430960 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5tc5v\"" Nov 01 00:23:31.431513 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.431497 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 00:23:31.431608 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.431552 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8q2z5\"" Nov 01 00:23:31.432102 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.432080 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 00:23:31.432207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.432193 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 00:23:31.432302 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.432266 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 00:23:31.432302 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.432219 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cx8lz\"" Nov 01 00:23:31.432908 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.432867 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9sbx\"" Nov 01 00:23:31.432993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.432928 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c54w\"" Nov 01 00:23:31.433156 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.433134 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 00:23:31.433253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.433237 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 00:23:31.433313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.433299 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rhkgk\"" Nov 01 00:23:31.433357 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.433174 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7ckpz\"" Nov 01 00:23:31.433780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.433761 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-km6f6\"" Nov 01 00:23:31.433933 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.433782 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 00:23:31.434023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.433802 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lrql\"" Nov 01 00:23:31.434168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.434036 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t74k4\"" Nov 01 00:23:31.434257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.434240 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 00:23:31.434306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.434267 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 00:23:31.434583 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.434565 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kfxmw\"" Nov 01 00:23:31.434583 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.434575 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqdjj\"" Nov 01 00:23:31.434814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.434780 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m6gx8\"" Nov 01 00:23:31.434932 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.434914 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k669s\"" Nov 01 00:23:31.435049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.435031 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwm8b\"" Nov 01 00:23:31.435113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.435058 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rjfbh\"" Nov 01 00:23:31.435178 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.435163 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctdk\"" Nov 01 00:23:31.435292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.435274 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2ck8t\"" Nov 01 00:23:31.438998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.438978 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.439073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439009 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd6c\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.439073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439027 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-ovn\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.439073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439054 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thc6l\" (UniqueName: \"kubernetes.io/projected/5cfa57d0-764b-42fb-ad36-9ca6dc472766-kube-api-access-thc6l\") pod \"kas-connection-checker-685895c7c9-fxsjd\" (UID: \"5cfa57d0-764b-42fb-ad36-9ca6dc472766\") " pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" Nov 01 00:23:31.439196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439097 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.439196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439150 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-var-lib-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.439291 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439262 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-log-socket\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.439340 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439322 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knxln\" (UniqueName: \"kubernetes.io/projected/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-kube-api-access-knxln\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:23:31.439406 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439393 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nngm4\" (UniqueName: \"kubernetes.io/projected/2508f7ff-f9da-48fe-8e1c-ba02568e0899-kube-api-access-nngm4\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.439514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439500 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.439618 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439603 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-device-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.439701 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439668 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzfvp\" (UniqueName: \"kubernetes.io/projected/b7e4c8da-dd72-438d-bd5b-a81728051a95-kube-api-access-dzfvp\") pod \"kas-connection-checker-685895c7c9-gsr76\" (UID: \"b7e4c8da-dd72-438d-bd5b-a81728051a95\") " pod="kube-system/kas-connection-checker-685895c7c9-gsr76" Nov 01 00:23:31.439771 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439757 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/783ad43d-a72e-4b58-b403-00eab40f6cdc-serving-cert\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:31.439815 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439801 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d98af5-fb06-47ca-a462-8f974bbfd046-tmp-dir\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.439878 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439865 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-lib-modules\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.439910 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439865 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d98af5-fb06-47ca-a462-8f974bbfd046-tmp-dir\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.439940 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439907 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac18260-596d-49f0-b109-d5750e8c3644-serving-cert\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.439940 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439929 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-etc-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.440000 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439947 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5twnl\" (UniqueName: \"kubernetes.io/projected/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-api-access-5twnl\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.440000 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439970 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/783ad43d-a72e-4b58-b403-00eab40f6cdc-serving-cert\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:31.440064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440030 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac18260-596d-49f0-b109-d5750e8c3644-serving-cert\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.440064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.439973 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.440125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440073 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-netd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.440160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440125 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhr6\" (UniqueName: \"kubernetes.io/projected/c767a0e3-268a-423f-8113-f98ab65fa4b3-kube-api-access-4hhr6\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.440160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440144 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.440220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440158 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:31.440220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440212 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwpgz\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.440344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440322 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.440435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440354 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.440435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440373 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzdn\" (UniqueName: \"kubernetes.io/projected/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-kube-api-access-jdzdn\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:31.440435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440390 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8m6\" (UniqueName: \"kubernetes.io/projected/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-kube-api-access-dh8m6\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:31.440435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440393 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:31.440435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440409 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.440722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440448 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-system-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.440722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440482 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.440722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440514 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-bin\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.440722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440573 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3246fb55-62ae-4d36-a951-75206d4f99d1-crio-socket\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.440722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440641 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghrw\" (UniqueName: \"kubernetes.io/projected/252a2753-2639-4920-afaf-0fb0ca316d26-kube-api-access-wghrw\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.440722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440675 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-socket-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440729 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-multus\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440730 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440776 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-tmp\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440809 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-config\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440837 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-tmp\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440837 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-kubelet\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440848 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440779 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.440969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.440938 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-kubelet\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441016 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-config\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441023 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-serviceca\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441051 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/252a2753-2639-4920-afaf-0fb0ca316d26-serving-cert\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441067 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-stats-auth\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441083 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-host\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441111 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7vb\" (UniqueName: \"kubernetes.io/projected/0ac18260-596d-49f0-b109-d5750e8c3644-kube-api-access-8k7vb\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441148 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-system-cni-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441173 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/252a2753-2639-4920-afaf-0fb0ca316d26-serving-cert\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441182 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441211 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-netns\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441218 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-serviceca\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:23:31.441297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441296 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-conf-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441322 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5fs\" (UniqueName: \"kubernetes.io/projected/9d259950-9b3c-41dc-87bd-74286a0acda0-kube-api-access-fh5fs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441300 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-stats-auth\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441366 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-default-certificate\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441473 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-run\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441505 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441597 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-registration-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441642 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsw6b\" (UniqueName: \"kubernetes.io/projected/0d85dc68-06e8-4036-93b1-ececa1a779c7-kube-api-access-lsw6b\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441670 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-default-certificate\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtb9\" (UniqueName: \"kubernetes.io/projected/5dc70a14-ff5e-4088-ab14-15700a98ad22-kube-api-access-cjtb9\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:31.441722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441720 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441726 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441737 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441763 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-script-lib\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441797 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441839 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae8165f7-bad3-418a-be14-97acb4604cf7-host-slash\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441868 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6l2\" (UniqueName: \"kubernetes.io/projected/38761a0d-2c71-4f3b-8d5d-39cdf321763f-kube-api-access-2j6l2\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441893 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/783ad43d-a72e-4b58-b403-00eab40f6cdc-config\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441923 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2b6\" (UniqueName: \"kubernetes.io/projected/48f9765c-72a2-4a67-b161-9de71454b8cc-kube-api-access-pc2b6\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441951 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-systemd-units\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441973 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovnkube-script-lib\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441974 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.441976 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4cab1f4-47e1-457c-a182-692b3be9bd16-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442016 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442043 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d98af5-fb06-47ca-a462-8f974bbfd046-config-volume\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442069 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92wl\" (UniqueName: \"kubernetes.io/projected/a9d98af5-fb06-47ca-a462-8f974bbfd046-kube-api-access-t92wl\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442096 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/783ad43d-a72e-4b58-b403-00eab40f6cdc-config\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442126 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442169 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-etc-kubernetes\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442200 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rnk\" (UniqueName: \"kubernetes.io/projected/8b062870-dd5d-4f35-b400-ea17ebd11878-kube-api-access-t6rnk\") pod \"network-check-source-6f78d59f99-vhjfm\" (UID: \"8b062870-dd5d-4f35-b400-ea17ebd11878\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442217 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqnt\" (UniqueName: \"kubernetes.io/projected/783ad43d-a72e-4b58-b403-00eab40f6cdc-kube-api-access-vhqnt\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442232 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442234 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4cab1f4-47e1-457c-a182-692b3be9bd16-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442245 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d98af5-fb06-47ca-a462-8f974bbfd046-config-volume\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.442255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442253 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-kubernetes\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442329 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442373 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442420 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3246fb55-62ae-4d36-a951-75206d4f99d1-data-volume\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442493 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-multus-certs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442509 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-var-lib-kubelet\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-tmp\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442555 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3246fb55-62ae-4d36-a951-75206d4f99d1-data-volume\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442563 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c767a0e3-268a-423f-8113-f98ab65fa4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442580 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-konnectivity-ca\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442602 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpc6\" (UniqueName: \"kubernetes.io/projected/c79db355-84dd-4f0a-b102-d7c28b1caee6-kube-api-access-vjpc6\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442586 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-tmp\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442634 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-modprobe-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442659 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-service-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442728 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-kubelet-config\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442775 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-node-log\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442786 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac18260-596d-49f0-b109-d5750e8c3644-service-ca-bundle\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442803 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-config\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442826 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-socket-dir-parent\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442842 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-conf\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442856 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-host\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442888 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442924 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-konnectivity-ca\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.442957 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-sys-fs\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443015 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-config\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443014 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce5465ef-bfd2-4075-8560-a19c26720d7e-hosts-file\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443058 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-tuned\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443117 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5xs\" (UniqueName: \"kubernetes.io/projected/c5b7952d-24a8-40c5-a755-3aeb8269714b-kube-api-access-hh5xs\") pod \"kas-connection-checker-685895c7c9-7btpl\" (UID: \"c5b7952d-24a8-40c5-a755-3aeb8269714b\") " pod="kube-system/kas-connection-checker-685895c7c9-7btpl" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443122 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443148 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-hostroot\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443125 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-tuned\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443173 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443200 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-env-overrides\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443237 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443315 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.443331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443341 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443365 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c767a0e3-268a-423f-8113-f98ab65fa4b3-env-overrides\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443359 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgbt\" (UniqueName: \"kubernetes.io/projected/3b775de6-474a-4551-8ddc-ade7a8c25fab-kube-api-access-hsgbt\") pod \"migrator-66df5d496b-6dzb5\" (UID: \"3b775de6-474a-4551-8ddc-ade7a8c25fab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443407 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-cert\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443430 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443518 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443537 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-sys\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443555 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-agent-certs\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443577 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443611 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae8165f7-bad3-418a-be14-97acb4604cf7-iptables-alerter-script\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443644 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-cert\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443666 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-cni-binary-copy\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443698 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcznv\" (UniqueName: \"kubernetes.io/projected/80541015-4203-4621-bba9-d737aa2f3266-kube-api-access-tcznv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443724 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443750 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-slash\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443769 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443773 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443818 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60389df0-3bb0-42fd-9a5b-40bfa22bacf5-agent-certs\") pod \"konnectivity-agent-vtb29\" (UID: \"60389df0-3bb0-42fd-9a5b-40bfa22bacf5\") " pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443822 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443833 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cni-binary-copy\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443858 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443889 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443931 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae8165f7-bad3-418a-be14-97acb4604cf7-iptables-alerter-script\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443994 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-cni-binary-copy\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.443996 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444032 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4454de50-086b-4842-8e00-d2ffae8344d2-original-pull-secret\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444040 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-daemon-config\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444073 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-systemd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444078 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444109 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444130 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cnibin\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444148 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gc8\" (UniqueName: \"kubernetes.io/projected/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc-kube-api-access-v9gc8\") pod \"volume-data-source-validator-58c9f5f46d-7x6c4\" (UID: \"70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444151 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-daemon-config\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444166 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444186 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-dbus\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444211 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-netns\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444282 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80541015-4203-4621-bba9-d737aa2f3266-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444312 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80541015-4203-4621-bba9-d737aa2f3266-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444345 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444356 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444484 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80541015-4203-4621-bba9-d737aa2f3266-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444466 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444503 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80541015-4203-4621-bba9-d737aa2f3266-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444512 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tgh\" (UniqueName: \"kubernetes.io/projected/ce5465ef-bfd2-4075-8560-a19c26720d7e-kube-api-access-d6tgh\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444598 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-cnibin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444614 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysconfig\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444646 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-snapshots\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444668 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7x8\" (UniqueName: \"kubernetes.io/projected/ae8165f7-bad3-418a-be14-97acb4604cf7-kube-api-access-rc7x8\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444692 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444698 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0ac18260-596d-49f0-b109-d5750e8c3644-snapshots\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444743 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-bin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444760 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444766 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444797 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444826 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5465ef-bfd2-4075-8560-a19c26720d7e-tmp-dir\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444845 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-k8s-cni-cncf-io\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444886 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-systemd\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.444862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444890 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5465ef-bfd2-4075-8560-a19c26720d7e-tmp-dir\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:23:31.446518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.444916 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:31.478214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.478187 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 00:23:31.478214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.478188 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 00:23:31.483787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.483767 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/252a2753-2639-4920-afaf-0fb0ca316d26-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.485644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.485612 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.485698 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.485651 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.500916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.500889 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knxln\" (UniqueName: \"kubernetes.io/projected/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-kube-api-access-knxln\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:23:31.501030 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.500952 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzfvp\" (UniqueName: \"kubernetes.io/projected/b7e4c8da-dd72-438d-bd5b-a81728051a95-kube-api-access-dzfvp\") pod \"kas-connection-checker-685895c7c9-gsr76\" (UID: \"b7e4c8da-dd72-438d-bd5b-a81728051a95\") " pod="kube-system/kas-connection-checker-685895c7c9-gsr76" Nov 01 00:23:31.501087 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501039 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngm4\" (UniqueName: \"kubernetes.io/projected/2508f7ff-f9da-48fe-8e1c-ba02568e0899-kube-api-access-nngm4\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.501087 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501069 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thc6l\" (UniqueName: \"kubernetes.io/projected/5cfa57d0-764b-42fb-ad36-9ca6dc472766-kube-api-access-thc6l\") pod \"kas-connection-checker-685895c7c9-fxsjd\" (UID: \"5cfa57d0-764b-42fb-ad36-9ca6dc472766\") " pod="kube-system/kas-connection-checker-685895c7c9-fxsjd" Nov 01 00:23:31.501251 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501229 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5twnl\" (UniqueName: \"kubernetes.io/projected/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-api-access-5twnl\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.501334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501318 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztd6c\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.501583 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501561 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghrw\" (UniqueName: \"kubernetes.io/projected/252a2753-2639-4920-afaf-0fb0ca316d26-kube-api-access-wghrw\") pod \"console-operator-7cd56b8dbc-nq7f9\" (UID: \"252a2753-2639-4920-afaf-0fb0ca316d26\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:31.501721 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501704 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhr6\" (UniqueName: \"kubernetes.io/projected/c767a0e3-268a-423f-8113-f98ab65fa4b3-kube-api-access-4hhr6\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.501955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501938 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwpgz\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.502014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.501951 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8m6\" (UniqueName: \"kubernetes.io/projected/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-kube-api-access-dh8m6\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:31.502109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.502093 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzdn\" (UniqueName: \"kubernetes.io/projected/62e97e00-f916-46ea-bcb1-2f0f52ebe2e5-kube-api-access-jdzdn\") pod \"ingress-canary-5hhd5\" (UID: \"62e97e00-f916-46ea-bcb1-2f0f52ebe2e5\") " pod="openshift-ingress-canary/ingress-canary-5hhd5" Nov 01 00:23:31.505533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.505510 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.506804 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.506779 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5fs\" (UniqueName: \"kubernetes.io/projected/9d259950-9b3c-41dc-87bd-74286a0acda0-kube-api-access-fh5fs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.506901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.506867 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7vb\" (UniqueName: \"kubernetes.io/projected/0ac18260-596d-49f0-b109-d5750e8c3644-kube-api-access-8k7vb\") pod \"insights-operator-7c4986d449-srh5n\" (UID: \"0ac18260-596d-49f0-b109-d5750e8c3644\") " pod="openshift-insights/insights-operator-7c4986d449-srh5n" Nov 01 00:23:31.506956 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.506920 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqnt\" (UniqueName: \"kubernetes.io/projected/783ad43d-a72e-4b58-b403-00eab40f6cdc-kube-api-access-vhqnt\") pod \"service-ca-operator-544d749c8c-ddtqb\" (UID: \"783ad43d-a72e-4b58-b403-00eab40f6cdc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" Nov 01 00:23:31.506991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.506977 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.507129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.507113 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtb9\" (UniqueName: \"kubernetes.io/projected/5dc70a14-ff5e-4088-ab14-15700a98ad22-kube-api-access-cjtb9\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:31.514167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.514148 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6l2\" (UniqueName: \"kubernetes.io/projected/38761a0d-2c71-4f3b-8d5d-39cdf321763f-kube-api-access-2j6l2\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:31.514255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.514165 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgbt\" (UniqueName: \"kubernetes.io/projected/3b775de6-474a-4551-8ddc-ade7a8c25fab-kube-api-access-hsgbt\") pod \"migrator-66df5d496b-6dzb5\" (UID: \"3b775de6-474a-4551-8ddc-ade7a8c25fab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" Nov 01 00:23:31.514319 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.514304 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsw6b\" (UniqueName: \"kubernetes.io/projected/0d85dc68-06e8-4036-93b1-ececa1a779c7-kube-api-access-lsw6b\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.514450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.514430 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rnk\" (UniqueName: \"kubernetes.io/projected/8b062870-dd5d-4f35-b400-ea17ebd11878-kube-api-access-t6rnk\") pod \"network-check-source-6f78d59f99-vhjfm\" (UID: \"8b062870-dd5d-4f35-b400-ea17ebd11878\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vhjfm" Nov 01 00:23:31.514540 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.514526 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc5n\" (UniqueName: \"kubernetes.io/projected/579dfa33-9adb-4613-b055-11671039eea2-kube-api-access-xqc5n\") pod \"network-check-target-bq86m\" (UID: \"579dfa33-9adb-4613-b055-11671039eea2\") " pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:31.519389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519311 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92wl\" (UniqueName: \"kubernetes.io/projected/a9d98af5-fb06-47ca-a462-8f974bbfd046-kube-api-access-t92wl\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.519389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519320 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcznv\" (UniqueName: \"kubernetes.io/projected/80541015-4203-4621-bba9-d737aa2f3266-kube-api-access-tcznv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5\" (UID: \"80541015-4203-4621-bba9-d737aa2f3266\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" Nov 01 00:23:31.519543 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519411 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5xs\" (UniqueName: \"kubernetes.io/projected/c5b7952d-24a8-40c5-a755-3aeb8269714b-kube-api-access-hh5xs\") pod \"kas-connection-checker-685895c7c9-7btpl\" (UID: \"c5b7952d-24a8-40c5-a755-3aeb8269714b\") " pod="kube-system/kas-connection-checker-685895c7c9-7btpl" Nov 01 00:23:31.519543 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519425 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7x8\" (UniqueName: \"kubernetes.io/projected/ae8165f7-bad3-418a-be14-97acb4604cf7-kube-api-access-rc7x8\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:23:31.519543 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519458 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tgh\" (UniqueName: \"kubernetes.io/projected/ce5465ef-bfd2-4075-8560-a19c26720d7e-kube-api-access-d6tgh\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:23:31.519543 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519523 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpc6\" (UniqueName: \"kubernetes.io/projected/c79db355-84dd-4f0a-b102-d7c28b1caee6-kube-api-access-vjpc6\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.519762 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519575 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gc8\" (UniqueName: \"kubernetes.io/projected/70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc-kube-api-access-v9gc8\") pod \"volume-data-source-validator-58c9f5f46d-7x6c4\" (UID: \"70f6bed4-8a11-4ecb-b334-9f76c5d9ddcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7x6c4" Nov 01 00:23:31.519762 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.519708 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2b6\" (UniqueName: \"kubernetes.io/projected/48f9765c-72a2-4a67-b161-9de71454b8cc-kube-api-access-pc2b6\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.546893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.546873 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-conf-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.546979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.546901 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-run\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.546979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.546917 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-registration-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.546979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.546942 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.546979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.546969 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae8165f7-bad3-418a-be14-97acb4604cf7-host-slash\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.546982 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-conf-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.546994 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-systemd-units\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547018 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547028 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae8165f7-bad3-418a-be14-97acb4604cf7-host-slash\") pod \"iptables-alerter-gwksj\" (UID: \"ae8165f7-bad3-418a-be14-97acb4604cf7\") " pod="openshift-network-operator/iptables-alerter-gwksj" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547027 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-registration-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547028 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-run\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547047 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-etc-kubernetes\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547065 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-systemd-units\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547084 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.047063182 +0000 UTC m=+2.026645501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547093 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-etc-kubernetes\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547102 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547106 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547133 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-kubernetes\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547155 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547178 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-multus-certs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547203 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-var-lib-kubelet\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547180 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547218 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-kubernetes\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547231 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-modprobe-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547247 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-var-lib-kubelet\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547205 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547211 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-multus-certs\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547268 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.047253711 +0000 UTC m=+2.026836047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547293 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-kubelet-config\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547309 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-node-log\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547325 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-socket-dir-parent\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547347 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-conf\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547324 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-modprobe-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547378 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-kubelet-config\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547386 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-multus-socket-dir-parent\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547373 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-host\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547410 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-node-log\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547422 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-sys-fs\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547439 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce5465ef-bfd2-4075-8560-a19c26720d7e-hosts-file\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547446 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4ba53d-6c1a-4032-b83d-225bf88e02e8-host\") pod \"node-ca-kl4cn\" (UID: \"9c4ba53d-6c1a-4032-b83d-225bf88e02e8\") " pod="openshift-image-registry/node-ca-kl4cn" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547455 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-hostroot\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547480 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547485 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-sys-fs\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547486 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-conf\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547507 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547526 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce5465ef-bfd2-4075-8560-a19c26720d7e-hosts-file\") pod \"node-resolver-v47fq\" (UID: \"ce5465ef-bfd2-4075-8560-a19c26720d7e\") " pod="openshift-dns/node-resolver-v47fq" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547559 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547557 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547585 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-hostroot\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547582 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547606 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-sys\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547644 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-slash\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547667 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547671 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-sys\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547682 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547695 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547699 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-slash\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547710 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547758 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547699 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547764 4910 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547770 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:31.547765 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547786 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.047771765 +0000 UTC m=+2.027354093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547823 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.047814002 +0000 UTC m=+2.027396331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : secret "insights-runtime-extractor-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547841 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.047832645 +0000 UTC m=+2.027414961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547864 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547897 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-systemd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547921 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547946 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cnibin\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.547986 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547988 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-systemd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547991 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-cnibin\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.547986 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548007 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-dbus\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548022 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.048008777 +0000 UTC m=+2.027591094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548044 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3246fb55-62ae-4d36-a951-75206d4f99d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548054 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-netns\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548055 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4454de50-086b-4842-8e00-d2ffae8344d2-dbus\") pod \"global-pull-secret-syncer-m45bc\" (UID: \"4454de50-086b-4842-8e00-d2ffae8344d2\") " pod="kube-system/global-pull-secret-syncer-m45bc" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548087 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-run-netns\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548124 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548139 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-cnibin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548153 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysconfig\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548170 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-bin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548185 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548186 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-cnibin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548202 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-k8s-cni-cncf-io\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548212 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548228 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548228 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-bin\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548241 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-systemd\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548217 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-systemd\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548268 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.048253722 +0000 UTC m=+2.027836067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548269 4910 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548286 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysconfig\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548284 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-k8s-cni-cncf-io\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548306 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548329 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548346 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.048333361 +0000 UTC m=+2.027915676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : secret "metrics-daemon-secret" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548364 4910 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548377 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-ovn\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548405 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-etc-sysctl-d\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548403 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-var-lib-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548414 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.048400763 +0000 UTC m=+2.027983099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-run-ovn\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548436 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-var-lib-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548439 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-log-socket\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548467 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-log-socket\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548469 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548517 4910 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548539 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-device-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548560 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-lib-modules\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548579 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-etc-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548593 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-device-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548603 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-netd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548648 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548652 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-netd\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548657 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-etc-openvswitch\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:31.548677 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:32.048660819 +0000 UTC m=+2.028243150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548695 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-lib-modules\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548701 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-system-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548738 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-system-cni-dir\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548748 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-bin\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548764 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3246fb55-62ae-4d36-a951-75206d4f99d1-crio-socket\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548779 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-socket-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548800 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-multus\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548832 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-cni-multus\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548843 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3246fb55-62ae-4d36-a951-75206d4f99d1-crio-socket\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548866 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-kubelet\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548870 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c79db355-84dd-4f0a-b102-d7c28b1caee6-socket-dir\") pod \"aws-ebs-csi-driver-node-6rk6v\" (UID: \"c79db355-84dd-4f0a-b102-d7c28b1caee6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rk6v" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548867 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-cni-bin\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548895 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-kubelet\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548915 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-var-lib-kubelet\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548924 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-host\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548948 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-system-cni-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548971 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a0e3-268a-423f-8113-f98ab65fa4b3-host-kubelet\") pod \"ovnkube-node-wg9vt\" (UID: \"c767a0e3-268a-423f-8113-f98ab65fa4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548974 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-netns\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.548976 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48f9765c-72a2-4a67-b161-9de71454b8cc-host\") pod \"tuned-g77tg\" (UID: \"48f9765c-72a2-4a67-b161-9de71454b8cc\") " pod="openshift-cluster-node-tuning-operator/tuned-g77tg" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.549007 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2508f7ff-f9da-48fe-8e1c-ba02568e0899-system-cni-dir\") pod \"multus-additional-cni-plugins-6qv8g\" (UID: \"2508f7ff-f9da-48fe-8e1c-ba02568e0899\") " pod="openshift-multus/multus-additional-cni-plugins-6qv8g" Nov 01 00:23:31.549313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.549019 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d259950-9b3c-41dc-87bd-74286a0acda0-host-run-netns\") pod \"multus-f66hm\" (UID: \"9d259950-9b3c-41dc-87bd-74286a0acda0\") " pod="openshift-multus/multus-f66hm" Nov 01 00:23:31.577699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.577678 4910 scope.go:122] "RemoveContainer" containerID="1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd" Nov 01 00:23:31.590584 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:31.590569 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:23:32.053109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053076 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:32.053109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053110 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053136 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053157 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053228 4910 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053228 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053241 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053260 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053278 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053262798 +0000 UTC m=+3.032845112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : secret "metrics-daemon-secret" not found Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053278 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053295 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053284044 +0000 UTC m=+3.032866394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:32.053304 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053237 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053327 4910 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053332 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.05332573 +0000 UTC m=+3.032908042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053344 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053338703 +0000 UTC m=+3.032921015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053355 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053350316 +0000 UTC m=+3.032932629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053380 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053428 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053453 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053486 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.053507 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053519 4910 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053525 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053551 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053538175 +0000 UTC m=+3.033120491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053565 4910 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053571 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053560593 +0000 UTC m=+3.033142913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053584 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053576708 +0000 UTC m=+3.033159021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053595 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053599 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.05359047 +0000 UTC m=+3.033172791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : secret "insights-runtime-extractor-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053605 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:32.053664 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.053651 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:33.053644141 +0000 UTC m=+3.033226453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:32.268360 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.268333 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:23:32.268770 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.268714 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/1.log" Nov 01 00:23:32.268770 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.268744 4910 generic.go:356] "Generic (PLEG): container finished" podID="252a2753-2639-4920-afaf-0fb0ca316d26" containerID="04330d35be15ac188b04e719203ac6341275525f98abeeae7f1153b88e2a988d" exitCode=255 Nov 01 00:23:32.268860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.268826 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" event={"ID":"252a2753-2639-4920-afaf-0fb0ca316d26","Type":"ContainerDied","Data":"04330d35be15ac188b04e719203ac6341275525f98abeeae7f1153b88e2a988d"} Nov 01 00:23:32.268911 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.268896 4910 scope.go:122] "RemoveContainer" containerID="1bddaa9c3d16bf358c922c02f2abe22110ea1f34b48e32e6572098003215abcd" Nov 01 00:23:32.269363 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.269347 4910 scope.go:122] "RemoveContainer" containerID="04330d35be15ac188b04e719203ac6341275525f98abeeae7f1153b88e2a988d" Nov 01 00:23:32.269543 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:32.269508 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq7f9_openshift-console-operator(252a2753-2639-4920-afaf-0fb0ca316d26)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" podUID="252a2753-2639-4920-afaf-0fb0ca316d26" Nov 01 00:23:32.768429 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.768397 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:23:32.905618 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:32.905542 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:33.059636 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.059599 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:33.059837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.059675 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:33.059837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.059738 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:33.059837 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059768 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.059735661 +0000 UTC m=+5.039317973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:33.059837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.059809 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:33.059837 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059817 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:33.059837 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059822 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:33.059837 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059837 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.059840 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059865 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.059853354 +0000 UTC m=+5.039435670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.059888 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059897 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.059881186 +0000 UTC m=+5.039463529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059901 4910 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059908 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.059939 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059953 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.059942773 +0000 UTC m=+5.039525085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059972 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059991 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.059984539 +0000 UTC m=+5.039566852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : secret "insights-runtime-extractor-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.059997 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060011 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.060020 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060025 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.060013414 +0000 UTC m=+5.039595743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060042 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.060036491 +0000 UTC m=+5.039618804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.060060 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060073 4910 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.060089 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:33.060110 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060097 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.060087596 +0000 UTC m=+5.039669945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : secret "metrics-daemon-secret" not found Nov 01 00:23:33.060708 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060128 4910 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:33.060708 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060149 4910 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:33.060708 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060160 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.06014475 +0000 UTC m=+5.039727066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:33.060708 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.060179 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:35.060172256 +0000 UTC m=+5.039754570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:33.273843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.273815 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:23:33.275341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.275320 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6dzb5" event={"ID":"3b775de6-474a-4551-8ddc-ade7a8c25fab","Type":"ContainerStarted","Data":"90cdbbd676801f65b9bfd5a35a1d03b3ba02d691c78560297b273465a8ed0eb8"} Nov 01 00:23:33.481047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.481015 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:33.481381 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:33.481368 4910 scope.go:122] "RemoveContainer" containerID="04330d35be15ac188b04e719203ac6341275525f98abeeae7f1153b88e2a988d" Nov 01 00:23:33.481549 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:33.481531 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq7f9_openshift-console-operator(252a2753-2639-4920-afaf-0fb0ca316d26)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" podUID="252a2753-2639-4920-afaf-0fb0ca316d26" Nov 01 00:23:35.079761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079718 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:35.079761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079763 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079800 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079824 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079850 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079869 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.079872 4910 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079888 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.079913 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.079942 4910 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.079996 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.079998 4910 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080034 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080042 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.079997 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.079949 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls podName:1ba5db8b-e910-4ba0-b604-d1a8b393d1f7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.079929205 +0000 UTC m=+9.059511520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2vfwz" (UID: "1ba5db8b-e910-4ba0-b604-d1a8b393d1f7") : secret "cluster-monitoring-operator-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080046 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.080147 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080186 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080170191 +0000 UTC m=+9.059752505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080196 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080205 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:35.080204 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080229 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080220236 +0000 UTC m=+9.059802581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080243 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls podName:a9d98af5-fb06-47ca-a462-8f974bbfd046 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.08023675 +0000 UTC m=+9.059819063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls") pod "dns-default-bvzrd" (UID: "a9d98af5-fb06-47ca-a462-8f974bbfd046") : secret "dns-default-metrics-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080255 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080249195 +0000 UTC m=+9.059831509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080262 4910 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080265 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls podName:3246fb55-62ae-4d36-a951-75206d4f99d1 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080260099 +0000 UTC m=+9.059842412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2s97" (UID: "3246fb55-62ae-4d36-a951-75206d4f99d1") : secret "insights-runtime-extractor-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080275 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080270125 +0000 UTC m=+9.059852438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080284 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080279698 +0000 UTC m=+9.059862011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080293 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080288694 +0000 UTC m=+9.059871006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:35.080284 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:35.080303 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs podName:38761a0d-2c71-4f3b-8d5d-39cdf321763f nodeName:}" failed. No retries permitted until 2026-11-01 00:23:39.080298995 +0000 UTC m=+9.059881308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs") pod "network-metrics-daemon-dtpjl" (UID: "38761a0d-2c71-4f3b-8d5d-39cdf321763f") : secret "metrics-daemon-secret" not found Nov 01 00:23:37.231288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:37.231261 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v47fq_ce5465ef-bfd2-4075-8560-a19c26720d7e/dns-node-resolver/0.log" Nov 01 00:23:37.831682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:37.831617 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kl4cn_9c4ba53d-6c1a-4032-b83d-225bf88e02e8/node-ca/0.log" Nov 01 00:23:38.631490 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:38.631457 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hhd5_62e97e00-f916-46ea-bcb1-2f0f52ebe2e5/serve-healthcheck-canary/0.log" Nov 01 00:23:39.120686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.120642 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.120704 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.120737 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120768 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:47.120752309 +0000 UTC m=+17.100334622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : configmap references non-existent config key: service-ca.crt Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.120802 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120836 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120839 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.120847 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:39.120859 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120858 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649fb6d9c5-llkn8: secret "image-registry-tls" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.120872 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120901 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs podName:0d85dc68-06e8-4036-93b1-ececa1a779c7 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:47.120887035 +0000 UTC m=+17.100469372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs") pod "router-default-69cbfc858f-rq59s" (UID: "0d85dc68-06e8-4036-93b1-ececa1a779c7") : secret "router-metrics-certs-default" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120922 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls podName:98642830-5eb4-42bf-8c4a-7ffabdc5d965 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:47.120912973 +0000 UTC m=+17.100495306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls") pod "image-registry-649fb6d9c5-llkn8" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965") : secret "image-registry-tls" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120934 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.120959 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120974 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert podName:b4cab1f4-47e1-457c-a182-692b3be9bd16 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:47.120960674 +0000 UTC m=+17.100543019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7vncn" (UID: "b4cab1f4-47e1-457c-a182-692b3be9bd16") : secret "networking-console-plugin-cert" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.120987 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.121007 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.121021 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls podName:5dc70a14-ff5e-4088-ab14-15700a98ad22 nodeName:}" failed. No retries permitted until 2026-11-01 00:23:47.121009925 +0000 UTC m=+17.100592239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7kjds" (UID: "5dc70a14-ff5e-4088-ab14-15700a98ad22") : secret "samples-operator-tls" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.121021 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.121035 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f6fdc666-mgj5x: secret "image-registry-tls" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.121053 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:39.121064 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls podName:0316e070-2f84-4652-82ce-51837acdc06e nodeName:}" failed. No retries permitted until 2026-11-01 00:23:47.121054341 +0000 UTC m=+17.100636674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls") pod "image-registry-78f6fdc666-mgj5x" (UID: "0316e070-2f84-4652-82ce-51837acdc06e") : secret "image-registry-tls" not found Nov 01 00:23:39.121192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.121098 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:39.121997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.121218 4910 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 00:23:39.123493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.123468 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3246fb55-62ae-4d36-a951-75206d4f99d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2s97\" (UID: \"3246fb55-62ae-4d36-a951-75206d4f99d1\") " pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:39.123608 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.123567 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9d98af5-fb06-47ca-a462-8f974bbfd046-metrics-tls\") pod \"dns-default-bvzrd\" (UID: \"a9d98af5-fb06-47ca-a462-8f974bbfd046\") " pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:39.123608 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.123569 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38761a0d-2c71-4f3b-8d5d-39cdf321763f-metrics-certs\") pod \"network-metrics-daemon-dtpjl\" (UID: \"38761a0d-2c71-4f3b-8d5d-39cdf321763f\") " pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:39.123851 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.123831 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ba5db8b-e910-4ba0-b604-d1a8b393d1f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2vfwz\" (UID: \"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:39.233306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.233274 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6dzb5_3b775de6-474a-4551-8ddc-ade7a8c25fab/migrator/0.log" Nov 01 00:23:39.378205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.378132 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" Nov 01 00:23:39.384938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.384920 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtpjl" Nov 01 00:23:39.388700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.388671 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:39.389706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.389691 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k2s97" Nov 01 00:23:39.441891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.441510 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5_80541015-4203-4621-bba9-d737aa2f3266/kube-storage-version-migrator-operator/0.log" Nov 01 00:23:39.707528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.707450 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:39.707907 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.707666 4910 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 00:23:39.797567 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:39.797539 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba5db8b_e910_4ba0_b604_d1a8b393d1f7.slice/crio-fbe38f21f4caf1f6f2b470e971aa832be9bbf85610d0925c1354eb9d24ce1c07 WatchSource:0}: Error finding container fbe38f21f4caf1f6f2b470e971aa832be9bbf85610d0925c1354eb9d24ce1c07: Status 404 returned error can't find the container with id fbe38f21f4caf1f6f2b470e971aa832be9bbf85610d0925c1354eb9d24ce1c07 Nov 01 00:23:39.811239 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:39.811200 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d98af5_fb06_47ca_a462_8f974bbfd046.slice/crio-ae0235521de57571c0d1c1674c2c8c07b99291d44b401de2f9d7ac28d41c578f WatchSource:0}: Error finding container ae0235521de57571c0d1c1674c2c8c07b99291d44b401de2f9d7ac28d41c578f: Status 404 returned error can't find the container with id ae0235521de57571c0d1c1674c2c8c07b99291d44b401de2f9d7ac28d41c578f Nov 01 00:23:39.811565 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.811547 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz"] Nov 01 00:23:39.847348 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:39.847323 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38761a0d_2c71_4f3b_8d5d_39cdf321763f.slice/crio-8685ba78aadee8515883d172955fea06f904f6e5b1116e539ccea34f9ddedb95 WatchSource:0}: Error finding container 8685ba78aadee8515883d172955fea06f904f6e5b1116e539ccea34f9ddedb95: Status 404 returned error can't find the container with id 8685ba78aadee8515883d172955fea06f904f6e5b1116e539ccea34f9ddedb95 Nov 01 00:23:39.847959 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:39.847895 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3246fb55_62ae_4d36_a951_75206d4f99d1.slice/crio-d1cb2e43b186716b272a56e8e0bc1748a43d742b1afc4d0258aba331c0251613 WatchSource:0}: Error finding container d1cb2e43b186716b272a56e8e0bc1748a43d742b1afc4d0258aba331c0251613: Status 404 returned error can't find the container with id d1cb2e43b186716b272a56e8e0bc1748a43d742b1afc4d0258aba331c0251613 Nov 01 00:23:39.865074 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.865031 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bvzrd"] Nov 01 00:23:39.865182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.865154 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dtpjl"] Nov 01 00:23:39.865236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:39.865193 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k2s97"] Nov 01 00:23:40.297547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:40.297506 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2s97" event={"ID":"3246fb55-62ae-4d36-a951-75206d4f99d1","Type":"ContainerStarted","Data":"0217e4aecbc4822eac789e1ca757a30bc5a43f8f0c0fcd4a74d5cdf365cafe20"} Nov 01 00:23:40.297851 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:40.297580 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2s97" event={"ID":"3246fb55-62ae-4d36-a951-75206d4f99d1","Type":"ContainerStarted","Data":"d1cb2e43b186716b272a56e8e0bc1748a43d742b1afc4d0258aba331c0251613"} Nov 01 00:23:40.298665 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:40.298618 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtpjl" event={"ID":"38761a0d-2c71-4f3b-8d5d-39cdf321763f","Type":"ContainerStarted","Data":"8685ba78aadee8515883d172955fea06f904f6e5b1116e539ccea34f9ddedb95"} Nov 01 00:23:40.299851 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:40.299822 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bvzrd" event={"ID":"a9d98af5-fb06-47ca-a462-8f974bbfd046","Type":"ContainerStarted","Data":"ae0235521de57571c0d1c1674c2c8c07b99291d44b401de2f9d7ac28d41c578f"} Nov 01 00:23:40.300928 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:40.300902 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" event={"ID":"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7","Type":"ContainerStarted","Data":"fbe38f21f4caf1f6f2b470e971aa832be9bbf85610d0925c1354eb9d24ce1c07"} Nov 01 00:23:40.932002 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:40.931975 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wg9vt" Nov 01 00:23:41.309216 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:41.309182 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2s97" event={"ID":"3246fb55-62ae-4d36-a951-75206d4f99d1","Type":"ContainerStarted","Data":"a88e35c42492ad4935c6466950e0676a7f9013be97aba7d37a829cf97345a1cb"} Nov 01 00:23:41.374906 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:41.374870 4910 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:41.375391 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:41.375361 4910 scope.go:122] "RemoveContainer" containerID="04330d35be15ac188b04e719203ac6341275525f98abeeae7f1153b88e2a988d" Nov 01 00:23:41.375589 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:41.375568 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq7f9_openshift-console-operator(252a2753-2639-4920-afaf-0fb0ca316d26)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" podUID="252a2753-2639-4920-afaf-0fb0ca316d26" Nov 01 00:23:42.315662 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:42.315601 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtpjl" event={"ID":"38761a0d-2c71-4f3b-8d5d-39cdf321763f","Type":"ContainerStarted","Data":"fd9ba6e4560fdb6a9cd022c3fb8874adb3dd44e5527325e5228e33963a862848"} Nov 01 00:23:42.316025 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:42.315698 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtpjl" event={"ID":"38761a0d-2c71-4f3b-8d5d-39cdf321763f","Type":"ContainerStarted","Data":"0b4667a6046d5adeb6b821023fededfaac291235c1eed0b549bceec276c10ac9"} Nov 01 00:23:42.317676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:42.317649 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bvzrd" event={"ID":"a9d98af5-fb06-47ca-a462-8f974bbfd046","Type":"ContainerStarted","Data":"adef72701568c0aaca6ec6c2ab9852b19902d418645d46dc476def9d5a3d3c32"} Nov 01 00:23:42.317930 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:42.317917 4910 scope.go:122] "RemoveContainer" containerID="04330d35be15ac188b04e719203ac6341275525f98abeeae7f1153b88e2a988d" Nov 01 00:23:43.323577 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.323536 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2s97" event={"ID":"3246fb55-62ae-4d36-a951-75206d4f99d1","Type":"ContainerStarted","Data":"c989d5a36a1ad34878d6dd8df04ceda81acc1536646d610dcce16f20edf5fc47"} Nov 01 00:23:43.325439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.325416 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bvzrd" event={"ID":"a9d98af5-fb06-47ca-a462-8f974bbfd046","Type":"ContainerStarted","Data":"61db983cc42d2c4b3f2ae906aae8b088f03f1e758ae086eb7ec809a261ca951b"} Nov 01 00:23:43.325569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.325476 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:43.327143 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.327119 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2vfwz" event={"ID":"1ba5db8b-e910-4ba0-b604-d1a8b393d1f7","Type":"ContainerStarted","Data":"308244fd9062036c7cafb4e6070a4d1ccf698e0b14d9159ef8f077bc7eed71f8"} Nov 01 00:23:43.328988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.328973 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:23:43.329100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.329078 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" event={"ID":"252a2753-2639-4920-afaf-0fb0ca316d26","Type":"ContainerStarted","Data":"f83e01d811b4e2485b082c23b10f6897a6b9c21ace40fd1f5cfbd93ca0215752"} Nov 01 00:23:43.329336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.329314 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:43.334315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:43.334296 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq7f9" Nov 01 00:23:46.739037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:46.739008 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f6fdc666-mgj5x"] Nov 01 00:23:46.739407 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:23:46.739258 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" podUID="0316e070-2f84-4652-82ce-51837acdc06e" Nov 01 00:23:47.195905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.195872 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:47.195905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.195913 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:47.196163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.195944 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:47.196163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.195979 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:47.196163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.196031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:47.196163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.196066 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:47.196712 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.196663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d85dc68-06e8-4036-93b1-ececa1a779c7-service-ca-bundle\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:47.198614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.198587 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"image-registry-649fb6d9c5-llkn8\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:47.199120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.199102 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d85dc68-06e8-4036-93b1-ececa1a779c7-metrics-certs\") pod \"router-default-69cbfc858f-rq59s\" (UID: \"0d85dc68-06e8-4036-93b1-ececa1a779c7\") " pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:47.199230 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.199210 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5dc70a14-ff5e-4088-ab14-15700a98ad22-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7kjds\" (UID: \"5dc70a14-ff5e-4088-ab14-15700a98ad22\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:47.199287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.199211 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4cab1f4-47e1-457c-a182-692b3be9bd16-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7vncn\" (UID: \"b4cab1f4-47e1-457c-a182-692b3be9bd16\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:47.199566 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.199550 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"image-registry-78f6fdc666-mgj5x\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:47.343386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.343359 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:47.349696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.349669 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:47.397077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397045 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397081 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397118 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397175 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397225 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397257 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397471 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397297 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397471 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397333 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") pod \"0316e070-2f84-4652-82ce-51837acdc06e\" (UID: \"0316e070-2f84-4652-82ce-51837acdc06e\") " Nov 01 00:23:47.397582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397549 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:23:47.397662 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397611 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:23:47.397776 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.397739 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:23:47.399691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.399663 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:23:47.399793 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.399712 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:23:47.399793 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.399781 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:23:47.400054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.400039 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "kube-api-access-dwpgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:23:47.400097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.400049 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls" pod "0316e070-2f84-4652-82ce-51837acdc06e" (UID: "0316e070-2f84-4652-82ce-51837acdc06e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:23:47.476694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.476616 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" Nov 01 00:23:47.477770 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.477740 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:47.478715 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.478323 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" Nov 01 00:23:47.488162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.488097 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:47.498529 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498501 4910 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-trusted-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.498673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498532 4910 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0316e070-2f84-4652-82ce-51837acdc06e-registry-certificates\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.498673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498551 4910 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0316e070-2f84-4652-82ce-51837acdc06e-ca-trust-extracted\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.498673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498567 4910 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-image-registry-private-configuration\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.498673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498582 4910 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-registry-tls\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.498673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498595 4910 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-bound-sa-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.498673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498608 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwpgz\" (UniqueName: \"kubernetes.io/projected/0316e070-2f84-4652-82ce-51837acdc06e-kube-api-access-dwpgz\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.498673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.498644 4910 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0316e070-2f84-4652-82ce-51837acdc06e-installation-pull-secrets\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:23:47.973362 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.973271 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7vncn"] Nov 01 00:23:47.975864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:47.975843 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds"] Nov 01 00:23:47.976405 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:47.976377 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cab1f4_47e1_457c_a182_692b3be9bd16.slice/crio-a878fefc891e9bbd0c8a6c544798cbca012838ee6914e02e0661015680cb5e55 WatchSource:0}: Error finding container a878fefc891e9bbd0c8a6c544798cbca012838ee6914e02e0661015680cb5e55: Status 404 returned error can't find the container with id a878fefc891e9bbd0c8a6c544798cbca012838ee6914e02e0661015680cb5e55 Nov 01 00:23:48.023129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.023096 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69cbfc858f-rq59s"] Nov 01 00:23:48.025570 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:48.025541 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d85dc68_06e8_4036_93b1_ececa1a779c7.slice/crio-bbb674833145970de4cd89e576bf8a8aca92844bf290a081fbb8e2393d53fee4 WatchSource:0}: Error finding container bbb674833145970de4cd89e576bf8a8aca92844bf290a081fbb8e2393d53fee4: Status 404 returned error can't find the container with id bbb674833145970de4cd89e576bf8a8aca92844bf290a081fbb8e2393d53fee4 Nov 01 00:23:48.027287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.027268 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-649fb6d9c5-llkn8"] Nov 01 00:23:48.030156 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:48.030137 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98642830_5eb4_42bf_8c4a_7ffabdc5d965.slice/crio-fae3464c1839eaa08b0899b7c69a2736d8692b83a4cd519aeb57a1d9c41dbbb2 WatchSource:0}: Error finding container fae3464c1839eaa08b0899b7c69a2736d8692b83a4cd519aeb57a1d9c41dbbb2: Status 404 returned error can't find the container with id fae3464c1839eaa08b0899b7c69a2736d8692b83a4cd519aeb57a1d9c41dbbb2 Nov 01 00:23:48.348773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.348735 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" event={"ID":"b4cab1f4-47e1-457c-a182-692b3be9bd16","Type":"ContainerStarted","Data":"a878fefc891e9bbd0c8a6c544798cbca012838ee6914e02e0661015680cb5e55"} Nov 01 00:23:48.351032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.350477 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" event={"ID":"98642830-5eb4-42bf-8c4a-7ffabdc5d965","Type":"ContainerStarted","Data":"53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7"} Nov 01 00:23:48.351032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.350564 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" event={"ID":"98642830-5eb4-42bf-8c4a-7ffabdc5d965","Type":"ContainerStarted","Data":"fae3464c1839eaa08b0899b7c69a2736d8692b83a4cd519aeb57a1d9c41dbbb2"} Nov 01 00:23:48.351032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.350684 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:23:48.351824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.351801 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" event={"ID":"5dc70a14-ff5e-4088-ab14-15700a98ad22","Type":"ContainerStarted","Data":"af49785f1106fe68c4fcd19e3eab9dac2b42272867bfd5ecca6d99ca7beb1bd4"} Nov 01 00:23:48.353348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.353327 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f6fdc666-mgj5x" Nov 01 00:23:48.353442 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.353359 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69cbfc858f-rq59s" event={"ID":"0d85dc68-06e8-4036-93b1-ececa1a779c7","Type":"ContainerStarted","Data":"ef704a3aba3de467f73970c79e441a7106b1b4f493d50f6680c87c3e5c0477bd"} Nov 01 00:23:48.353442 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.353397 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69cbfc858f-rq59s" event={"ID":"0d85dc68-06e8-4036-93b1-ececa1a779c7","Type":"ContainerStarted","Data":"bbb674833145970de4cd89e576bf8a8aca92844bf290a081fbb8e2393d53fee4"} Nov 01 00:23:48.478038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.478006 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:48.480903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.480871 4910 patch_prober.go:29] interesting pod/router-default-69cbfc858f-rq59s container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 01 00:23:48.480903 ip-10-0-137-150 kubenswrapper[4910]: [-]has-synced failed: reason withheld Nov 01 00:23:48.480903 ip-10-0-137-150 kubenswrapper[4910]: [+]process-running ok Nov 01 00:23:48.480903 ip-10-0-137-150 kubenswrapper[4910]: healthz check failed Nov 01 00:23:48.481111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.480923 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-69cbfc858f-rq59s" podUID="0d85dc68-06e8-4036-93b1-ececa1a779c7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 01 00:23:48.772134 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.772102 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f6fdc666-mgj5x"] Nov 01 00:23:48.777914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:48.777885 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78f6fdc666-mgj5x"] Nov 01 00:23:49.480524 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:49.480498 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:50.190295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:50.190261 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0316e070-2f84-4652-82ce-51837acdc06e" path="/var/lib/kubelet/pods/0316e070-2f84-4652-82ce-51837acdc06e/volumes" Nov 01 00:23:50.361940 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:50.361905 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7vncn" event={"ID":"b4cab1f4-47e1-457c-a182-692b3be9bd16","Type":"ContainerStarted","Data":"0761c20907b8e5252290fe422f172c472897e394114ddf51aa02d1c9c8593620"} Nov 01 00:23:50.362104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:50.362093 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:50.363275 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:50.363251 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69cbfc858f-rq59s" Nov 01 00:23:51.368428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:51.368385 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" event={"ID":"5dc70a14-ff5e-4088-ab14-15700a98ad22","Type":"ContainerStarted","Data":"4e1a784fc91f135d539afe69f801cff87498e3450574908f868d4404400fbafe"} Nov 01 00:23:51.368939 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:51.368447 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7kjds" event={"ID":"5dc70a14-ff5e-4088-ab14-15700a98ad22","Type":"ContainerStarted","Data":"700505b02c663c44a471037acd218c9065f37d8cb939c30f7f39640bb80e10bc"} Nov 01 00:23:53.335225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:53.335194 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bvzrd" Nov 01 00:23:56.128887 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:56.128863 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtb29" Nov 01 00:23:57.250111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.250076 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q4xhx"] Nov 01 00:23:57.285863 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.285835 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.337980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.337947 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 00:23:57.338293 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.338271 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vngfn\"" Nov 01 00:23:57.338369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.338341 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 00:23:57.379864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.379838 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-wtmp\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.379870 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-textfile\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.379887 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-tls\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.379949 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-root\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.380088 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.380129 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmdr\" (UniqueName: \"kubernetes.io/projected/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-kube-api-access-fxmdr\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.380172 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-metrics-client-ca\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.380205 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.380372 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.380270 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-sys\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.423274 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.423248 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 00:23:57.423398 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.423310 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 00:23:57.481421 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481393 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-wtmp\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481421 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481422 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-textfile\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481445 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-tls\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481472 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-root\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481490 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481510 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmdr\" (UniqueName: \"kubernetes.io/projected/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-kube-api-access-fxmdr\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481587 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-wtmp\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481661 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-metrics-client-ca\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481709 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481747 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-sys\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481655 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-root\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481845 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-sys\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.481919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.481845 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-textfile\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.482226 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.482132 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.482226 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.482205 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-metrics-client-ca\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.484065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.484035 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-tls\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.484065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.484050 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.542173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.542108 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmdr\" (UniqueName: \"kubernetes.io/projected/4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c-kube-api-access-fxmdr\") pod \"node-exporter-q4xhx\" (UID: \"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c\") " pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.636333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:57.636299 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4xhx" Nov 01 00:23:57.667606 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:23:57.667574 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0bb0df_b83c_4a00_975d_fdf9b2ffb50c.slice/crio-89504f29fac943807f66e4be5142c6d90b1b233d0be852a1771f045af46c37f8 WatchSource:0}: Error finding container 89504f29fac943807f66e4be5142c6d90b1b233d0be852a1771f045af46c37f8: Status 404 returned error can't find the container with id 89504f29fac943807f66e4be5142c6d90b1b233d0be852a1771f045af46c37f8 Nov 01 00:23:58.399458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:23:58.399416 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4xhx" event={"ID":"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c","Type":"ContainerStarted","Data":"89504f29fac943807f66e4be5142c6d90b1b233d0be852a1771f045af46c37f8"} Nov 01 00:24:00.408023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:00.407973 4910 generic.go:356] "Generic (PLEG): container finished" podID="4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c" containerID="af0f5baa052fe0459518dfecd2a06c0c1776aa0cd92ebd3af8cae0dd30621ca8" exitCode=0 Nov 01 00:24:00.408400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:00.408059 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4xhx" event={"ID":"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c","Type":"ContainerDied","Data":"af0f5baa052fe0459518dfecd2a06c0c1776aa0cd92ebd3af8cae0dd30621ca8"} Nov 01 00:24:01.413962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:01.413928 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4xhx" event={"ID":"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c","Type":"ContainerStarted","Data":"c8a7b2083679870da990024b94b3f92b69fa276c8a6fac65c2a317baa6cfde19"} Nov 01 00:24:01.414358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:01.413978 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4xhx" event={"ID":"4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c","Type":"ContainerStarted","Data":"a285c2ba5e7ef881ade901abc2b8fef5ae9368f2d64014168002834b0c5681e8"} Nov 01 00:24:01.534116 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:01.534065 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q4xhx" podStartSLOduration=2.5980873620000002 podStartE2EDuration="4.534051508s" totalImagesPullingTime="1.854431881s" totalInitContainerRuntime="81.532265ms" isStatefulPod=true podCreationTimestamp="2026-11-01 00:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:24:01.532973098 +0000 UTC m=+31.512555432" watchObservedRunningTime="2026-11-01 00:24:01.534051508 +0000 UTC m=+31.513633842" Nov 01 00:24:02.771509 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:02.771482 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bq86m" Nov 01 00:24:08.879892 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:08.879812 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-649fb6d9c5-llkn8"] Nov 01 00:24:08.884458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:08.884435 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:24:13.103119 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:24:13.103078 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:24:33.900459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:33.900412 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" podUID="98642830-5eb4-42bf-8c4a-7ffabdc5d965" containerName="registry" containerID="cri-o://53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7" gracePeriod=30 Nov 01 00:24:34.145263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.145240 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:24:34.165069 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165008 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165069 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165046 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165085 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165115 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165134 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165165 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165204 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165243 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\") pod \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\" (UID: \"98642830-5eb4-42bf-8c4a-7ffabdc5d965\") " Nov 01 00:24:34.165605 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165574 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:34.165711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.165689 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:24:34.167911 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.167879 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:34.168033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.167958 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "kube-api-access-ztd6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:34.168146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.168121 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:34.168306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.168201 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:24:34.168306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.168225 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:24:34.172268 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.172215 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted" pod "98642830-5eb4-42bf-8c4a-7ffabdc5d965" (UID: "98642830-5eb4-42bf-8c4a-7ffabdc5d965"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:24:34.265941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265906 4910 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-certificates\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.265941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265933 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztd6c\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-kube-api-access-ztd6c\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.265941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265944 4910 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-registry-tls\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.266177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265954 4910 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-installation-pull-secrets\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.266177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265962 4910 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98642830-5eb4-42bf-8c4a-7ffabdc5d965-trusted-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.266177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265972 4910 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98642830-5eb4-42bf-8c4a-7ffabdc5d965-image-registry-private-configuration\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.266177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265982 4910 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98642830-5eb4-42bf-8c4a-7ffabdc5d965-ca-trust-extracted\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.266177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.265992 4910 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98642830-5eb4-42bf-8c4a-7ffabdc5d965-bound-sa-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:24:34.542296 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.542265 4910 generic.go:356] "Generic (PLEG): container finished" podID="98642830-5eb4-42bf-8c4a-7ffabdc5d965" containerID="53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7" exitCode=0 Nov 01 00:24:34.542467 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.542306 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" event={"ID":"98642830-5eb4-42bf-8c4a-7ffabdc5d965","Type":"ContainerDied","Data":"53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7"} Nov 01 00:24:34.542467 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.542340 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" event={"ID":"98642830-5eb4-42bf-8c4a-7ffabdc5d965","Type":"ContainerDied","Data":"fae3464c1839eaa08b0899b7c69a2736d8692b83a4cd519aeb57a1d9c41dbbb2"} Nov 01 00:24:34.542467 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.542358 4910 scope.go:122] "RemoveContainer" containerID="53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7" Nov 01 00:24:34.542467 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.542308 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649fb6d9c5-llkn8" Nov 01 00:24:34.550297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.550278 4910 scope.go:122] "RemoveContainer" containerID="53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7" Nov 01 00:24:34.550573 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:24:34.550553 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7\": container with ID starting with 53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7 not found: ID does not exist" containerID="53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7" Nov 01 00:24:34.550680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.550583 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7"} err="failed to get container status \"53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7\": rpc error: code = NotFound desc = could not find container \"53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7\": container with ID starting with 53fbc340da760aba994b43870c796614f6f69479c5f723d932ed8b3b970e96d7 not found: ID does not exist" Nov 01 00:24:34.707502 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.707468 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-649fb6d9c5-llkn8"] Nov 01 00:24:34.711248 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:34.711229 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-649fb6d9c5-llkn8"] Nov 01 00:24:36.188784 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:36.188756 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98642830-5eb4-42bf-8c4a-7ffabdc5d965" path="/var/lib/kubelet/pods/98642830-5eb4-42bf-8c4a-7ffabdc5d965/volumes" Nov 01 00:24:43.101653 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:24:43.101600 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:24:46.587749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:46.587712 4910 generic.go:356] "Generic (PLEG): container finished" podID="0ac18260-596d-49f0-b109-d5750e8c3644" containerID="2ed4996273be3b24a329987ea56c8e86bf346535c294c4ca58f85096772af5a0" exitCode=0 Nov 01 00:24:46.588132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:46.587776 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-srh5n" event={"ID":"0ac18260-596d-49f0-b109-d5750e8c3644","Type":"ContainerDied","Data":"2ed4996273be3b24a329987ea56c8e86bf346535c294c4ca58f85096772af5a0"} Nov 01 00:24:46.588132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:46.588114 4910 scope.go:122] "RemoveContainer" containerID="2ed4996273be3b24a329987ea56c8e86bf346535c294c4ca58f85096772af5a0" Nov 01 00:24:47.593664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:47.593609 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-srh5n" event={"ID":"0ac18260-596d-49f0-b109-d5750e8c3644","Type":"ContainerStarted","Data":"005c59aef864c99e7b5e9ee6684f968709e7f1b1a4012965f3f78d5d44a2fd46"} Nov 01 00:24:51.608357 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:51.608320 4910 generic.go:356] "Generic (PLEG): container finished" podID="783ad43d-a72e-4b58-b403-00eab40f6cdc" containerID="fcae44f4cf72e9e0d532fc74152e4ed221c563a4251f5fabcd63eb88c28075ba" exitCode=0 Nov 01 00:24:51.608797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:51.608383 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" event={"ID":"783ad43d-a72e-4b58-b403-00eab40f6cdc","Type":"ContainerDied","Data":"fcae44f4cf72e9e0d532fc74152e4ed221c563a4251f5fabcd63eb88c28075ba"} Nov 01 00:24:51.608797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:51.608690 4910 scope.go:122] "RemoveContainer" containerID="fcae44f4cf72e9e0d532fc74152e4ed221c563a4251f5fabcd63eb88c28075ba" Nov 01 00:24:52.614051 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:52.614014 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ddtqb" event={"ID":"783ad43d-a72e-4b58-b403-00eab40f6cdc","Type":"ContainerStarted","Data":"c95c29212bd3e3c0bb2759676ca94db83a40c57cdf881b83aee02ac17e391d1e"} Nov 01 00:24:56.628893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:56.628861 4910 generic.go:356] "Generic (PLEG): container finished" podID="80541015-4203-4621-bba9-d737aa2f3266" containerID="b3d4bfd1b16cae4582f062644bfbfc307deb92d7446a87683e3559696ca14f1d" exitCode=0 Nov 01 00:24:56.629290 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:56.628940 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" event={"ID":"80541015-4203-4621-bba9-d737aa2f3266","Type":"ContainerDied","Data":"b3d4bfd1b16cae4582f062644bfbfc307deb92d7446a87683e3559696ca14f1d"} Nov 01 00:24:56.629290 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:56.629248 4910 scope.go:122] "RemoveContainer" containerID="b3d4bfd1b16cae4582f062644bfbfc307deb92d7446a87683e3559696ca14f1d" Nov 01 00:24:57.634091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:24:57.634060 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nbfv5" event={"ID":"80541015-4203-4621-bba9-d737aa2f3266","Type":"ContainerStarted","Data":"c47bedaa273d3fe1f26f84b7debb1b3351922620cb6fa12c1478a065f2d775cf"} Nov 01 00:25:13.102251 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:25:13.102207 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:25:43.101798 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:25:43.101718 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:26:13.101594 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:26:13.101553 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:26:43.101609 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:26:43.101561 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:27:13.101211 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:27:13.101174 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:27:43.101310 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:27:43.101273 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:28:13.101873 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:28:13.101831 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:28:30.212266 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:28:30.212239 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:28:30.213034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:28:30.213009 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:28:43.101291 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:28:43.101209 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:29:13.101754 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:29:13.101703 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:29:43.101923 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:29:43.101880 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:30:13.101565 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:30:13.101517 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:30:43.101481 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:30:43.101437 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:31:13.101708 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:31:13.101673 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:31:43.101595 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:31:43.101512 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:32:13.102274 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:32:13.102238 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:32:40.647615 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.647580 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc"] Nov 01 00:32:40.648287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.648191 4910 cpu_manager.go:414] "containerMap: removing container" podUID="98642830-5eb4-42bf-8c4a-7ffabdc5d965" containerName="registry" Nov 01 00:32:40.648287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.648223 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98642830-5eb4-42bf-8c4a-7ffabdc5d965" containerName="registry" Nov 01 00:32:40.651640 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.648705 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="98642830-5eb4-42bf-8c4a-7ffabdc5d965" containerName="registry" Nov 01 00:32:40.653853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.653829 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.708647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.708596 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 00:32:40.708647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.708609 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 00:32:40.758885 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.758857 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-kvr6j\"" Nov 01 00:32:40.759837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.759819 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc"] Nov 01 00:32:40.760555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.760537 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl"] Nov 01 00:32:40.763027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.763012 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.763072 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.763047 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lhf\" (UniqueName: \"kubernetes.io/projected/cef603bf-0e60-4882-bdfc-706a86af05ea-kube-api-access-l9lhf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.763107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.763075 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.763815 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.763801 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.816989 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.816947 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-fn58l\"" Nov 01 00:32:40.818144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.818122 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl"] Nov 01 00:32:40.864236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864206 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.864400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864262 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.864400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864298 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.864400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864325 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.864504 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864416 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbvc\" (UniqueName: \"kubernetes.io/projected/28944ecf-df2d-44c8-9d6e-49c2a803f813-kube-api-access-xkbvc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.864504 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864468 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lhf\" (UniqueName: \"kubernetes.io/projected/cef603bf-0e60-4882-bdfc-706a86af05ea-kube-api-access-l9lhf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.864683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.864725 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.864694 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.867482 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.867465 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2"] Nov 01 00:32:40.870778 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.870762 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:40.919524 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.919453 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2"] Nov 01 00:32:40.923391 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.923368 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-84dpl\"" Nov 01 00:32:40.926559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.926534 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lhf\" (UniqueName: \"kubernetes.io/projected/cef603bf-0e60-4882-bdfc-706a86af05ea-kube-api-access-l9lhf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.965286 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965252 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbvc\" (UniqueName: \"kubernetes.io/projected/28944ecf-df2d-44c8-9d6e-49c2a803f813-kube-api-access-xkbvc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.965456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965299 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:40.965456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965336 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xsg\" (UniqueName: \"kubernetes.io/projected/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-kube-api-access-95xsg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:40.965456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965361 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.965456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965381 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:40.965606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965515 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.965773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965755 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.965868 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.965851 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:40.968740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.968715 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:32:40.969053 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.969035 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt"] Nov 01 00:32:40.972142 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:40.972125 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.024257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.024234 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8mrx7\"" Nov 01 00:32:41.026849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.026817 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbvc\" (UniqueName: \"kubernetes.io/projected/28944ecf-df2d-44c8-9d6e-49c2a803f813-kube-api-access-xkbvc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:41.032179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.032059 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt"] Nov 01 00:32:41.066726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.066695 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:41.066893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.066733 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zbb\" (UniqueName: \"kubernetes.io/projected/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-kube-api-access-w2zbb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.066893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.066765 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xsg\" (UniqueName: \"kubernetes.io/projected/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-kube-api-access-95xsg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:41.066893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.066848 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:41.066893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.066889 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.067038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.066913 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.067131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.067114 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:41.067212 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.067196 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:41.069946 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.069921 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg"] Nov 01 00:32:41.073331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.073316 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.078510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.078489 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:32:41.128059 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.127703 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-jj5hp\"" Nov 01 00:32:41.133654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.129283 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xsg\" (UniqueName: \"kubernetes.io/projected/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-kube-api-access-95xsg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:41.133654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.133294 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg"] Nov 01 00:32:41.167680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.167643 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.167891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.167786 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zbb\" (UniqueName: \"kubernetes.io/projected/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-kube-api-access-w2zbb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.167965 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.167866 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.167965 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.167935 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mn8\" (UniqueName: \"kubernetes.io/projected/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-kube-api-access-v2mn8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.168070 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.168036 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.168126 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.168089 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.168550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.168524 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.168670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.168533 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.186035 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.185979 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:32:41.229239 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.229201 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zbb\" (UniqueName: \"kubernetes.io/projected/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-kube-api-access-w2zbb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.269179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.269148 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.269349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.269194 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mn8\" (UniqueName: \"kubernetes.io/projected/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-kube-api-access-v2mn8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.269349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.269266 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.269540 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.269519 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.269610 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.269588 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.300754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.300724 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:32:41.331044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.331014 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mn8\" (UniqueName: \"kubernetes.io/projected/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-kube-api-access-v2mn8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.341288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.341257 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc"] Nov 01 00:32:41.345116 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:41.345086 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef603bf_0e60_4882_bdfc_706a86af05ea.slice/crio-dba492f43fb8d2d6f623dbd58a5c44dc79a4485791fdb0aa3d9c84339505ea78 WatchSource:0}: Error finding container dba492f43fb8d2d6f623dbd58a5c44dc79a4485791fdb0aa3d9c84339505ea78: Status 404 returned error can't find the container with id dba492f43fb8d2d6f623dbd58a5c44dc79a4485791fdb0aa3d9c84339505ea78 Nov 01 00:32:41.347202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.347181 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:32:41.389772 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.389747 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:32:41.449328 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.449300 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl"] Nov 01 00:32:41.453572 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:41.453546 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28944ecf_df2d_44c8_9d6e_49c2a803f813.slice/crio-2fdc85834346fe3a1d17f3f29d038cba55868949c5d254404f77fd0267286534 WatchSource:0}: Error finding container 2fdc85834346fe3a1d17f3f29d038cba55868949c5d254404f77fd0267286534: Status 404 returned error can't find the container with id 2fdc85834346fe3a1d17f3f29d038cba55868949c5d254404f77fd0267286534 Nov 01 00:32:41.558845 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.558823 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2"] Nov 01 00:32:41.561463 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:41.561427 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae231ee1_1de0_4df6_8b1d_0c2e249098f3.slice/crio-860caef6cf5660917c19437e8bb6e019d22f97508fc672e144218b16bef0b3ef WatchSource:0}: Error finding container 860caef6cf5660917c19437e8bb6e019d22f97508fc672e144218b16bef0b3ef: Status 404 returned error can't find the container with id 860caef6cf5660917c19437e8bb6e019d22f97508fc672e144218b16bef0b3ef Nov 01 00:32:41.666053 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.666024 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6"] Nov 01 00:32:41.670843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.670824 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.673303 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.673279 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt"] Nov 01 00:32:41.675989 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:41.675963 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6de775b_8d31_41d8_8c59_cf2d36d17b3f.slice/crio-2eb51e39a0930ab3d5f8b8d6ef3a8963e94bd4b699c0d19bc2f049a6b64aaf98 WatchSource:0}: Error finding container 2eb51e39a0930ab3d5f8b8d6ef3a8963e94bd4b699c0d19bc2f049a6b64aaf98: Status 404 returned error can't find the container with id 2eb51e39a0930ab3d5f8b8d6ef3a8963e94bd4b699c0d19bc2f049a6b64aaf98 Nov 01 00:32:41.723434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.723361 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nnk4p\"" Nov 01 00:32:41.727453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.727426 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6"] Nov 01 00:32:41.760134 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.760112 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg"] Nov 01 00:32:41.762966 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:41.762934 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c4449a_f5d7_4ea1_a3e3_d1cee8ceea0c.slice/crio-5b855fd2be6cba5d69ef8d0f525a4d124c537e2ebfb73d4cb80cf31042ee5dcf WatchSource:0}: Error finding container 5b855fd2be6cba5d69ef8d0f525a4d124c537e2ebfb73d4cb80cf31042ee5dcf: Status 404 returned error can't find the container with id 5b855fd2be6cba5d69ef8d0f525a4d124c537e2ebfb73d4cb80cf31042ee5dcf Nov 01 00:32:41.773841 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.773822 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kb7\" (UniqueName: \"kubernetes.io/projected/868d4588-719a-49c2-8f30-d1d02663c0c5-kube-api-access-59kb7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.773912 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.773866 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.773912 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.773895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.874985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.874952 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kb7\" (UniqueName: \"kubernetes.io/projected/868d4588-719a-49c2-8f30-d1d02663c0c5-kube-api-access-59kb7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.875152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.875000 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.875152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.875028 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.875392 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.875375 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.875443 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.875410 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.936689 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.936655 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kb7\" (UniqueName: \"kubernetes.io/projected/868d4588-719a-49c2-8f30-d1d02663c0c5-kube-api-access-59kb7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:41.988536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:41.988440 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:32:42.253136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:42.253104 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" event={"ID":"a6de775b-8d31-41d8-8c59-cf2d36d17b3f","Type":"ContainerStarted","Data":"2eb51e39a0930ab3d5f8b8d6ef3a8963e94bd4b699c0d19bc2f049a6b64aaf98"} Nov 01 00:32:42.254962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:42.254932 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" event={"ID":"28944ecf-df2d-44c8-9d6e-49c2a803f813","Type":"ContainerStarted","Data":"2fdc85834346fe3a1d17f3f29d038cba55868949c5d254404f77fd0267286534"} Nov 01 00:32:42.257430 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:42.257395 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" event={"ID":"cef603bf-0e60-4882-bdfc-706a86af05ea","Type":"ContainerStarted","Data":"dba492f43fb8d2d6f623dbd58a5c44dc79a4485791fdb0aa3d9c84339505ea78"} Nov 01 00:32:42.262109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:42.262071 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" event={"ID":"ae231ee1-1de0-4df6-8b1d-0c2e249098f3","Type":"ContainerStarted","Data":"860caef6cf5660917c19437e8bb6e019d22f97508fc672e144218b16bef0b3ef"} Nov 01 00:32:42.263553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:42.263527 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" event={"ID":"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c","Type":"ContainerStarted","Data":"5b855fd2be6cba5d69ef8d0f525a4d124c537e2ebfb73d4cb80cf31042ee5dcf"} Nov 01 00:32:42.360817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:42.360778 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6"] Nov 01 00:32:42.364193 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:42.364163 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868d4588_719a_49c2_8f30_d1d02663c0c5.slice/crio-d94682c0ed5ee8ba166ad2332f22487ce5492a66314f7ce4e43d90c2c56ceddf WatchSource:0}: Error finding container d94682c0ed5ee8ba166ad2332f22487ce5492a66314f7ce4e43d90c2c56ceddf: Status 404 returned error can't find the container with id d94682c0ed5ee8ba166ad2332f22487ce5492a66314f7ce4e43d90c2c56ceddf Nov 01 00:32:43.118935 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:32:43.118857 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:32:43.280773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:43.280736 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" event={"ID":"868d4588-719a-49c2-8f30-d1d02663c0c5","Type":"ContainerStarted","Data":"d94682c0ed5ee8ba166ad2332f22487ce5492a66314f7ce4e43d90c2c56ceddf"} Nov 01 00:32:44.886142 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:44.886112 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt"] Nov 01 00:32:44.891008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:44.890050 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:44.995234 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:44.995017 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt"] Nov 01 00:32:44.995234 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:44.995144 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-7zbpz\"" Nov 01 00:32:45.007582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.007547 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.007860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.007693 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7f7\" (UniqueName: \"kubernetes.io/projected/d289c0f5-5d64-4638-8121-4ac3df4e36b0-kube-api-access-zd7f7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.007860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.007727 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.109505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.108358 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7f7\" (UniqueName: \"kubernetes.io/projected/d289c0f5-5d64-4638-8121-4ac3df4e36b0-kube-api-access-zd7f7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.109505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.108408 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.109505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.108444 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.109505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.108912 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.109505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.109449 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.182205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.182088 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7f7\" (UniqueName: \"kubernetes.io/projected/d289c0f5-5d64-4638-8121-4ac3df4e36b0-kube-api-access-zd7f7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.213808 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.213539 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:32:45.641637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:45.641584 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt"] Nov 01 00:32:45.652311 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:45.652278 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd289c0f5_5d64_4638_8121_4ac3df4e36b0.slice/crio-626faeac2c53e4fb66f7f7d2dc13e6c429bf6e64820eeb42db1e4807bce5e482 WatchSource:0}: Error finding container 626faeac2c53e4fb66f7f7d2dc13e6c429bf6e64820eeb42db1e4807bce5e482: Status 404 returned error can't find the container with id 626faeac2c53e4fb66f7f7d2dc13e6c429bf6e64820eeb42db1e4807bce5e482 Nov 01 00:32:46.309415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:46.309381 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" event={"ID":"d289c0f5-5d64-4638-8121-4ac3df4e36b0","Type":"ContainerStarted","Data":"626faeac2c53e4fb66f7f7d2dc13e6c429bf6e64820eeb42db1e4807bce5e482"} Nov 01 00:32:47.268129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.266549 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2"] Nov 01 00:32:47.270859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.270831 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.315677 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.315645 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6"] Nov 01 00:32:47.332316 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.331497 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.332316 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.331602 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrv7\" (UniqueName: \"kubernetes.io/projected/3bc446fe-d5bb-4080-9ae5-775a73321a58-kube-api-access-8wrv7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.332316 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.331763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.337235 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.336773 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.366318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.366237 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2"] Nov 01 00:32:47.367985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.367958 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn"] Nov 01 00:32:47.372571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.372514 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.384690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.384667 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6"] Nov 01 00:32:47.387733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.387708 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk"] Nov 01 00:32:47.392693 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.392676 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.424686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.424581 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn"] Nov 01 00:32:47.433295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433241 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.433438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433350 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.433438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433390 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5472\" (UniqueName: \"kubernetes.io/projected/7acaa955-c1c3-4640-aea4-8b659298da11-kube-api-access-k5472\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.433512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433457 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrv7\" (UniqueName: \"kubernetes.io/projected/3bc446fe-d5bb-4080-9ae5-775a73321a58-kube-api-access-8wrv7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.433512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433484 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvfj\" (UniqueName: \"kubernetes.io/projected/c25f286c-8837-4637-be75-1e1c0f611042-kube-api-access-cpvfj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.433578 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433550 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.433690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433674 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.433752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433702 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.433804 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.433782 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.434361 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.434025 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.434361 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.434144 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.451611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.451579 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk"] Nov 01 00:32:47.517596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.517558 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrv7\" (UniqueName: \"kubernetes.io/projected/3bc446fe-d5bb-4080-9ae5-775a73321a58-kube-api-access-8wrv7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.535068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.534993 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvfj\" (UniqueName: \"kubernetes.io/projected/c25f286c-8837-4637-be75-1e1c0f611042-kube-api-access-cpvfj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.535068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535044 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.535289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535085 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.535289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535124 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdmx\" (UniqueName: \"kubernetes.io/projected/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-kube-api-access-rtdmx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.535289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535164 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.535289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535205 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.535289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535248 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.535289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535279 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5472\" (UniqueName: \"kubernetes.io/projected/7acaa955-c1c3-4640-aea4-8b659298da11-kube-api-access-k5472\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.535592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535307 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.536009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.535982 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.536313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.536293 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.536589 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.536570 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.536863 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.536818 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.595170 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.594960 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:32:47.603198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.603128 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvfj\" (UniqueName: \"kubernetes.io/projected/c25f286c-8837-4637-be75-1e1c0f611042-kube-api-access-cpvfj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.603774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.603729 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5472\" (UniqueName: \"kubernetes.io/projected/7acaa955-c1c3-4640-aea4-8b659298da11-kube-api-access-k5472\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.637450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.636354 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.637450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.636445 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdmx\" (UniqueName: \"kubernetes.io/projected/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-kube-api-access-rtdmx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.637450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.636482 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.637450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.636908 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.637450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.637137 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.662245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.661946 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:32:47.697501 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.697435 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:32:47.708561 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.708532 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdmx\" (UniqueName: \"kubernetes.io/projected/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-kube-api-access-rtdmx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.717191 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.717162 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:32:47.835208 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.834569 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr"] Nov 01 00:32:47.845220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.840232 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.894468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.893602 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 00:32:47.894468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.893778 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhv5\"" Nov 01 00:32:47.903455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.903417 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr"] Nov 01 00:32:47.939144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939042 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939077 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939106 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mlj\" (UniqueName: \"kubernetes.io/projected/c16617ec-fda2-4f67-853b-fd58ff568d2c-kube-api-access-87mlj\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939177 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939302 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939362 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939391 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16617ec-fda2-4f67-853b-fd58ff568d2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939433 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:47.939555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:47.939466 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.024253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.024219 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2"] Nov 01 00:32:48.039966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.039931 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040042 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040065 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mlj\" (UniqueName: \"kubernetes.io/projected/c16617ec-fda2-4f67-853b-fd58ff568d2c-kube-api-access-87mlj\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040094 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040134 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040190 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040218 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16617ec-fda2-4f67-853b-fd58ff568d2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.040456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.040257 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.041314 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.041285 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.041558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.041538 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.041904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.041883 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.042175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.042115 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.042640 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.042599 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16617ec-fda2-4f67-853b-fd58ff568d2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.044841 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.044813 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.048680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.048573 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.053681 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:48.053592 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc446fe_d5bb_4080_9ae5_775a73321a58.slice/crio-0e32b5270e45d413b3450d6ffb316a50eb78b33e7fe155b00b31d5eb5170ea47 WatchSource:0}: Error finding container 0e32b5270e45d413b3450d6ffb316a50eb78b33e7fe155b00b31d5eb5170ea47: Status 404 returned error can't find the container with id 0e32b5270e45d413b3450d6ffb316a50eb78b33e7fe155b00b31d5eb5170ea47 Nov 01 00:32:48.103444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.102637 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6"] Nov 01 00:32:48.105501 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:48.105424 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acaa955_c1c3_4640_aea4_8b659298da11.slice/crio-0fe7073727bf1941aa0620464d6a1ca49e4aa120d4dbeb66eb87e0d44ab4d6a8 WatchSource:0}: Error finding container 0fe7073727bf1941aa0620464d6a1ca49e4aa120d4dbeb66eb87e0d44ab4d6a8: Status 404 returned error can't find the container with id 0fe7073727bf1941aa0620464d6a1ca49e4aa120d4dbeb66eb87e0d44ab4d6a8 Nov 01 00:32:48.112520 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.112224 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16617ec-fda2-4f67-853b-fd58ff568d2c-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.112520 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.112451 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mlj\" (UniqueName: \"kubernetes.io/projected/c16617ec-fda2-4f67-853b-fd58ff568d2c-kube-api-access-87mlj\") pod \"maas-default-gateway-openshift-default-67bf986f99-hg4hr\" (UID: \"c16617ec-fda2-4f67-853b-fd58ff568d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.135617 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.135577 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn"] Nov 01 00:32:48.145279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.145251 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw"] Nov 01 00:32:48.150954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.150935 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.168758 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.168733 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:32:48.200272 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.200246 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk"] Nov 01 00:32:48.205189 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.204940 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-pvwrb\"" Nov 01 00:32:48.205947 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:48.205912 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd272f3_4873_45e4_ac0d_2dfd9457fea5.slice/crio-1eae03030752bcf6b0fdb6f0e47cae2555303dd0772d3a82973cd41defd76c77 WatchSource:0}: Error finding container 1eae03030752bcf6b0fdb6f0e47cae2555303dd0772d3a82973cd41defd76c77: Status 404 returned error can't find the container with id 1eae03030752bcf6b0fdb6f0e47cae2555303dd0772d3a82973cd41defd76c77 Nov 01 00:32:48.216733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.216711 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw"] Nov 01 00:32:48.242434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242397 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.242585 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242483 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.242585 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242525 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.242759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242586 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.242759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242683 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.242759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242711 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bh4k\" (UniqueName: \"kubernetes.io/projected/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-kube-api-access-4bh4k\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.242915 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242824 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.242915 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242852 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.243021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.242937 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.344911 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.344959 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.345015 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.345059 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.345092 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.345126 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.345161 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.345187 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.345211 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bh4k\" (UniqueName: \"kubernetes.io/projected/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-kube-api-access-4bh4k\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.346009 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.346555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.346260 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.347524 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.347407 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.348053 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.348019 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.348884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.348774 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.349928 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.349885 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.354226 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.354165 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.367932 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.367869 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" event={"ID":"6dd272f3-4873-45e4-ac0d-2dfd9457fea5","Type":"ContainerStarted","Data":"1eae03030752bcf6b0fdb6f0e47cae2555303dd0772d3a82973cd41defd76c77"} Nov 01 00:32:48.371557 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.371498 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" event={"ID":"c25f286c-8837-4637-be75-1e1c0f611042","Type":"ContainerStarted","Data":"0f0ec715ba575bbea24a55258319b756a2036100bd7f711ebf95e2843210b48f"} Nov 01 00:32:48.382249 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.382218 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" event={"ID":"7acaa955-c1c3-4640-aea4-8b659298da11","Type":"ContainerStarted","Data":"0fe7073727bf1941aa0620464d6a1ca49e4aa120d4dbeb66eb87e0d44ab4d6a8"} Nov 01 00:32:48.389189 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.389123 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" event={"ID":"3bc446fe-d5bb-4080-9ae5-775a73321a58","Type":"ContainerStarted","Data":"0e32b5270e45d413b3450d6ffb316a50eb78b33e7fe155b00b31d5eb5170ea47"} Nov 01 00:32:48.410397 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.410324 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.417051 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.417016 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bh4k\" (UniqueName: \"kubernetes.io/projected/0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7-kube-api-access-4bh4k\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw\" (UID: \"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.470378 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.470344 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:32:48.630093 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:48.630053 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16617ec_fda2_4f67_853b_fd58ff568d2c.slice/crio-538c79cf95d42e4b09a9d826cc5c18672cf2f03f636758bf831c4185e617df6c WatchSource:0}: Error finding container 538c79cf95d42e4b09a9d826cc5c18672cf2f03f636758bf831c4185e617df6c: Status 404 returned error can't find the container with id 538c79cf95d42e4b09a9d826cc5c18672cf2f03f636758bf831c4185e617df6c Nov 01 00:32:48.630640 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.630592 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr"] Nov 01 00:32:48.916438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:48.916152 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw"] Nov 01 00:32:48.931677 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:32:48.931610 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0599ef9c_ba0c_4f5c_a5b6_8af53d4d1dd7.slice/crio-b0a920702a1faf1549268f2499cbff38705dcd94f3b0b392a684eb656e89780d WatchSource:0}: Error finding container b0a920702a1faf1549268f2499cbff38705dcd94f3b0b392a684eb656e89780d: Status 404 returned error can't find the container with id b0a920702a1faf1549268f2499cbff38705dcd94f3b0b392a684eb656e89780d Nov 01 00:32:49.411159 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:49.411113 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" event={"ID":"c16617ec-fda2-4f67-853b-fd58ff568d2c","Type":"ContainerStarted","Data":"538c79cf95d42e4b09a9d826cc5c18672cf2f03f636758bf831c4185e617df6c"} Nov 01 00:32:49.418372 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:32:49.418298 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" event={"ID":"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7","Type":"ContainerStarted","Data":"b0a920702a1faf1549268f2499cbff38705dcd94f3b0b392a684eb656e89780d"} Nov 01 00:33:02.399989 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:02.399937 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:33:02.400360 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:02.400038 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:33:02.400360 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:02.400078 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:33:02.605851 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:02.602600 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" event={"ID":"a6de775b-8d31-41d8-8c59-cf2d36d17b3f","Type":"ContainerStarted","Data":"6ae6c5679881414f90257e4d63a6de2ea368755e651524b169f5233d43d8554d"} Nov 01 00:33:02.606328 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:02.606302 4910 generic.go:356] "Generic (PLEG): container finished" podID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerID="53cd34888396a9e350bc15c55a7b173d70b1172c059cde99e0ca61f673444701" exitCode=0 Nov 01 00:33:02.607749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:02.606820 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" event={"ID":"cef603bf-0e60-4882-bdfc-706a86af05ea","Type":"ContainerDied","Data":"53cd34888396a9e350bc15c55a7b173d70b1172c059cde99e0ca61f673444701"} Nov 01 00:33:02.610500 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:02.609960 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" event={"ID":"c16617ec-fda2-4f67-853b-fd58ff568d2c","Type":"ContainerStarted","Data":"607f39218bae366afdb1e3fc9eb6466c33c6d59243d2a523d394be75b7311efe"} Nov 01 00:33:03.001979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.001925 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" podStartSLOduration=2.239660251 podStartE2EDuration="16.001909313s" totalImagesPullingTime="13.762249062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:02.999281268 +0000 UTC m=+572.978863603" watchObservedRunningTime="2026-11-01 00:33:03.001909313 +0000 UTC m=+572.981491648" Nov 01 00:33:03.170111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.170054 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:33:03.170111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.170109 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:33:03.175409 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.175385 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:33:03.618611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.618577 4910 generic.go:356] "Generic (PLEG): container finished" podID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerID="6ae6c5679881414f90257e4d63a6de2ea368755e651524b169f5233d43d8554d" exitCode=0 Nov 01 00:33:03.619032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.618664 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" event={"ID":"a6de775b-8d31-41d8-8c59-cf2d36d17b3f","Type":"ContainerDied","Data":"6ae6c5679881414f90257e4d63a6de2ea368755e651524b169f5233d43d8554d"} Nov 01 00:33:03.620459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.620379 4910 generic.go:356] "Generic (PLEG): container finished" podID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerID="d60e4e195996ba4220b1b3802c72a005df22c30c6851ab6dd1a8858fb2b34c6c" exitCode=0 Nov 01 00:33:03.620561 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.620461 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" event={"ID":"6dd272f3-4873-45e4-ac0d-2dfd9457fea5","Type":"ContainerDied","Data":"d60e4e195996ba4220b1b3802c72a005df22c30c6851ab6dd1a8858fb2b34c6c"} Nov 01 00:33:03.622432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.622409 4910 generic.go:356] "Generic (PLEG): container finished" podID="c25f286c-8837-4637-be75-1e1c0f611042" containerID="d540dd428f050d8c6281ff9141c4c88853bf752fb5c28aa6d964bcbcff51ff75" exitCode=0 Nov 01 00:33:03.622533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.622502 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" event={"ID":"c25f286c-8837-4637-be75-1e1c0f611042","Type":"ContainerDied","Data":"d540dd428f050d8c6281ff9141c4c88853bf752fb5c28aa6d964bcbcff51ff75"} Nov 01 00:33:03.624144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.624122 4910 generic.go:356] "Generic (PLEG): container finished" podID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerID="9659c9b35f2dc80d10bbcd5d4941a9875c6ef9d90ecc49fa7be029b4636cd1f8" exitCode=0 Nov 01 00:33:03.624239 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.624169 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" event={"ID":"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c","Type":"ContainerDied","Data":"9659c9b35f2dc80d10bbcd5d4941a9875c6ef9d90ecc49fa7be029b4636cd1f8"} Nov 01 00:33:03.625779 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.625757 4910 generic.go:356] "Generic (PLEG): container finished" podID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerID="974b3328e4d23cd69dd3915a92945c8a41d2afcfc445742edbaa2ad99b9708ef" exitCode=0 Nov 01 00:33:03.625874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.625838 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" event={"ID":"3bc446fe-d5bb-4080-9ae5-775a73321a58","Type":"ContainerDied","Data":"974b3328e4d23cd69dd3915a92945c8a41d2afcfc445742edbaa2ad99b9708ef"} Nov 01 00:33:03.627303 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.627275 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:33:03.685983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.685946 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:33:03.686100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.686042 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:33:03.686100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:03.686080 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:33:04.631898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.631820 4910 generic.go:356] "Generic (PLEG): container finished" podID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerID="7723bddf0c033ae73d761762bf4b9f1733a2160d9c3c90bd8c1c51bb85db0fec" exitCode=0 Nov 01 00:33:04.631898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.631890 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" event={"ID":"28944ecf-df2d-44c8-9d6e-49c2a803f813","Type":"ContainerDied","Data":"7723bddf0c033ae73d761762bf4b9f1733a2160d9c3c90bd8c1c51bb85db0fec"} Nov 01 00:33:04.633906 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.633878 4910 generic.go:356] "Generic (PLEG): container finished" podID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerID="207e286548f6032683aacec30d0ab9aabdf3240990609637d746e50e1b363c6c" exitCode=0 Nov 01 00:33:04.634029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.633961 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" event={"ID":"cef603bf-0e60-4882-bdfc-706a86af05ea","Type":"ContainerDied","Data":"207e286548f6032683aacec30d0ab9aabdf3240990609637d746e50e1b363c6c"} Nov 01 00:33:04.635481 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.635457 4910 generic.go:356] "Generic (PLEG): container finished" podID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerID="b0d49e71bb32e1107290f8f4af27a55c4057d7df53864f61ae0716ad8cc099c3" exitCode=0 Nov 01 00:33:04.635583 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.635529 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" event={"ID":"868d4588-719a-49c2-8f30-d1d02663c0c5","Type":"ContainerDied","Data":"b0d49e71bb32e1107290f8f4af27a55c4057d7df53864f61ae0716ad8cc099c3"} Nov 01 00:33:04.637177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.637156 4910 generic.go:356] "Generic (PLEG): container finished" podID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerID="7f932b8b4a489f00e26fcdd9551b3e44e5eb1a61218bf94c27ccae3cfded20c5" exitCode=0 Nov 01 00:33:04.637278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.637254 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" event={"ID":"d289c0f5-5d64-4638-8121-4ac3df4e36b0","Type":"ContainerDied","Data":"7f932b8b4a489f00e26fcdd9551b3e44e5eb1a61218bf94c27ccae3cfded20c5"} Nov 01 00:33:04.638818 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.638791 4910 generic.go:356] "Generic (PLEG): container finished" podID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerID="97ca2162aee2e59a4724d17db1c018ad9e52d13d6f500c6283ed8d27da081ded" exitCode=0 Nov 01 00:33:04.638885 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.638863 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" event={"ID":"ae231ee1-1de0-4df6-8b1d-0c2e249098f3","Type":"ContainerDied","Data":"97ca2162aee2e59a4724d17db1c018ad9e52d13d6f500c6283ed8d27da081ded"} Nov 01 00:33:04.640366 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.640341 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" event={"ID":"0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7","Type":"ContainerStarted","Data":"2cff9ff132dd797f2f8ca0d4ec301ed3f999ee5a81b5c45c783e7be9e28ec4cb"} Nov 01 00:33:04.641706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.641687 4910 generic.go:356] "Generic (PLEG): container finished" podID="7acaa955-c1c3-4640-aea4-8b659298da11" containerID="35f783decaf61358c258d18bc5290cd0d27c4c7f0c1ca39019d92190bf64d1b1" exitCode=0 Nov 01 00:33:04.641807 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:04.641765 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" event={"ID":"7acaa955-c1c3-4640-aea4-8b659298da11","Type":"ContainerDied","Data":"35f783decaf61358c258d18bc5290cd0d27c4c7f0c1ca39019d92190bf64d1b1"} Nov 01 00:33:05.103041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.102979 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" podStartSLOduration=2.363950067 podStartE2EDuration="17.102963268s" totalImagesPullingTime="14.739013201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:05.102768425 +0000 UTC m=+575.082350761" watchObservedRunningTime="2026-11-01 00:33:05.102963268 +0000 UTC m=+575.082545629" Nov 01 00:33:05.471136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.471112 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:33:05.475928 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.475904 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:33:05.648135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.648101 4910 generic.go:356] "Generic (PLEG): container finished" podID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerID="ce488b9a27566d002f9aefb688a1a4fb49be0616594e82897ffee7ff1ab8659c" exitCode=0 Nov 01 00:33:05.648552 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.648186 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" event={"ID":"a6de775b-8d31-41d8-8c59-cf2d36d17b3f","Type":"ContainerDied","Data":"ce488b9a27566d002f9aefb688a1a4fb49be0616594e82897ffee7ff1ab8659c"} Nov 01 00:33:05.650185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.650164 4910 generic.go:356] "Generic (PLEG): container finished" podID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerID="9e485e075ca7455a759c678ff4047be0bd86421a059494d66efc4aa7b33c99f8" exitCode=0 Nov 01 00:33:05.650288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.650250 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" event={"ID":"6dd272f3-4873-45e4-ac0d-2dfd9457fea5","Type":"ContainerDied","Data":"9e485e075ca7455a759c678ff4047be0bd86421a059494d66efc4aa7b33c99f8"} Nov 01 00:33:05.650592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.650569 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:33:05.651652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:05.651613 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw" Nov 01 00:33:06.661045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.660042 4910 generic.go:356] "Generic (PLEG): container finished" podID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerID="0bd9472bd2414c96a3d38a52c3deb3faf89ed44838861b461c5f5b5611e4c2f3" exitCode=0 Nov 01 00:33:06.661045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.660132 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" event={"ID":"28944ecf-df2d-44c8-9d6e-49c2a803f813","Type":"ContainerDied","Data":"0bd9472bd2414c96a3d38a52c3deb3faf89ed44838861b461c5f5b5611e4c2f3"} Nov 01 00:33:06.668269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.668235 4910 generic.go:356] "Generic (PLEG): container finished" podID="c25f286c-8837-4637-be75-1e1c0f611042" containerID="9ce9dc504ebece71fd344acf975edb93d30d9e12d1be7dda0e9b5de5ef29a3d9" exitCode=0 Nov 01 00:33:06.668456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.668358 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" event={"ID":"c25f286c-8837-4637-be75-1e1c0f611042","Type":"ContainerDied","Data":"9ce9dc504ebece71fd344acf975edb93d30d9e12d1be7dda0e9b5de5ef29a3d9"} Nov 01 00:33:06.675245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.675214 4910 generic.go:356] "Generic (PLEG): container finished" podID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerID="cf221b1f938eca45cefd41950798ee3504191a172503e3b1fa02de34788a8362" exitCode=0 Nov 01 00:33:06.675400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.675382 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" event={"ID":"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c","Type":"ContainerDied","Data":"cf221b1f938eca45cefd41950798ee3504191a172503e3b1fa02de34788a8362"} Nov 01 00:33:06.678819 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.678793 4910 generic.go:356] "Generic (PLEG): container finished" podID="7acaa955-c1c3-4640-aea4-8b659298da11" containerID="316de46e4a5214f41877f8b8fcd2f79c28519c7c02e59af855ad8cc0a3746239" exitCode=0 Nov 01 00:33:06.678901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.678883 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" event={"ID":"7acaa955-c1c3-4640-aea4-8b659298da11","Type":"ContainerDied","Data":"316de46e4a5214f41877f8b8fcd2f79c28519c7c02e59af855ad8cc0a3746239"} Nov 01 00:33:06.683349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.683326 4910 generic.go:356] "Generic (PLEG): container finished" podID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerID="a24a4fb6f36c6736adefe2fc945f9c7e798e3fd8fa97bf91297c3c38f3ccdeb0" exitCode=0 Nov 01 00:33:06.683986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:06.683917 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" event={"ID":"3bc446fe-d5bb-4080-9ae5-775a73321a58","Type":"ContainerDied","Data":"a24a4fb6f36c6736adefe2fc945f9c7e798e3fd8fa97bf91297c3c38f3ccdeb0"} Nov 01 00:33:07.706514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:07.706396 4910 generic.go:356] "Generic (PLEG): container finished" podID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerID="480a2048306864ec109d9f361fdf7a8783ebc166b8a3a4941f5292b7456cbd94" exitCode=0 Nov 01 00:33:07.707882 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:07.707854 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" event={"ID":"ae231ee1-1de0-4df6-8b1d-0c2e249098f3","Type":"ContainerDied","Data":"480a2048306864ec109d9f361fdf7a8783ebc166b8a3a4941f5292b7456cbd94"} Nov 01 00:33:10.760455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:10.760417 4910 generic.go:356] "Generic (PLEG): container finished" podID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerID="b741337924e12f79f0634e087db2152e15a2681aec1569c798f4e137db9c23c3" exitCode=0 Nov 01 00:33:10.761043 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:10.760573 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" event={"ID":"868d4588-719a-49c2-8f30-d1d02663c0c5","Type":"ContainerDied","Data":"b741337924e12f79f0634e087db2152e15a2681aec1569c798f4e137db9c23c3"} Nov 01 00:33:11.794811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:11.793908 4910 generic.go:356] "Generic (PLEG): container finished" podID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerID="4b21bf244786970d4a425de0a0827eae6b95b42ed4171fb7b7476e8b749e62f5" exitCode=0 Nov 01 00:33:11.794811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:11.793948 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" event={"ID":"d289c0f5-5d64-4638-8121-4ac3df4e36b0","Type":"ContainerDied","Data":"4b21bf244786970d4a425de0a0827eae6b95b42ed4171fb7b7476e8b749e62f5"} Nov 01 00:33:13.196597 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:33:13.196450 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:33:28.068714 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:28.068353 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" event={"ID":"6dd272f3-4873-45e4-ac0d-2dfd9457fea5","Type":"ContainerStarted","Data":"6c966c927474303d59a3b9bc2bd839756e0dfc42f2bbc598a72f030a6c9a0d61"} Nov 01 00:33:28.197428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:28.196765 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" podStartSLOduration=2.860361417 podStartE2EDuration="41.196750427s" totalImagesPullingTime="38.33638901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:28.195741425 +0000 UTC m=+598.175323778" watchObservedRunningTime="2026-11-01 00:33:28.196750427 +0000 UTC m=+598.176332766" Nov 01 00:33:29.096609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:29.096571 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" event={"ID":"a6de775b-8d31-41d8-8c59-cf2d36d17b3f","Type":"ContainerStarted","Data":"bd07115f06022d8e0672dc6bd185447d5031c61a853252d44d0b49bad9d16ac4"} Nov 01 00:33:29.103507 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:29.103403 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" event={"ID":"cef603bf-0e60-4882-bdfc-706a86af05ea","Type":"ContainerStarted","Data":"9e7dd457038ecb62ce148c5d6221ca0ffd9e524cab8199c03c0d56098880894d"} Nov 01 00:33:29.109991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:29.109963 4910 generic.go:356] "Generic (PLEG): container finished" podID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerID="6c966c927474303d59a3b9bc2bd839756e0dfc42f2bbc598a72f030a6c9a0d61" exitCode=0 Nov 01 00:33:29.110110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:29.110006 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" event={"ID":"6dd272f3-4873-45e4-ac0d-2dfd9457fea5","Type":"ContainerDied","Data":"6c966c927474303d59a3b9bc2bd839756e0dfc42f2bbc598a72f030a6c9a0d61"} Nov 01 00:33:29.217275 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:29.217171 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" podStartSLOduration=3.887458668 podStartE2EDuration="49.217151279s" totalImagesPullingTime="45.329692611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:29.214956045 +0000 UTC m=+599.194538397" watchObservedRunningTime="2026-11-01 00:33:29.217151279 +0000 UTC m=+599.196733616" Nov 01 00:33:29.336019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:29.335910 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" podStartSLOduration=3.798326381 podStartE2EDuration="49.335890311s" totalImagesPullingTime="45.53756393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:29.334133889 +0000 UTC m=+599.313716225" watchObservedRunningTime="2026-11-01 00:33:29.335890311 +0000 UTC m=+599.315472648" Nov 01 00:33:30.125306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.125265 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" event={"ID":"a6de775b-8d31-41d8-8c59-cf2d36d17b3f","Type":"ContainerDied","Data":"bd07115f06022d8e0672dc6bd185447d5031c61a853252d44d0b49bad9d16ac4"} Nov 01 00:33:30.125782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.125166 4910 generic.go:356] "Generic (PLEG): container finished" podID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerID="bd07115f06022d8e0672dc6bd185447d5031c61a853252d44d0b49bad9d16ac4" exitCode=0 Nov 01 00:33:30.128865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.128836 4910 generic.go:356] "Generic (PLEG): container finished" podID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerID="9e7dd457038ecb62ce148c5d6221ca0ffd9e524cab8199c03c0d56098880894d" exitCode=0 Nov 01 00:33:30.128997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.128935 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" event={"ID":"cef603bf-0e60-4882-bdfc-706a86af05ea","Type":"ContainerDied","Data":"9e7dd457038ecb62ce148c5d6221ca0ffd9e524cab8199c03c0d56098880894d"} Nov 01 00:33:30.133373 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.133351 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" event={"ID":"c25f286c-8837-4637-be75-1e1c0f611042","Type":"ContainerStarted","Data":"56ae5af20f8e22b4b3c09bc76b6ce9b0a36c599ea0979e6ac5575495d4ef484e"} Nov 01 00:33:30.342775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.342743 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:33:30.359821 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.359772 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:33:30.359992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.359975 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:33:30.395542 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.395507 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-bundle\") pod \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " Nov 01 00:33:30.395680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.395659 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-util\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-util\") pod \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " Nov 01 00:33:30.395757 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.395737 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-kube-api-access-rtdmx\" (UniqueName: \"kubernetes.io/projected/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-kube-api-access-rtdmx\") pod \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\" (UID: \"6dd272f3-4873-45e4-ac0d-2dfd9457fea5\") " Nov 01 00:33:30.397655 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.397612 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-bundle" pod "6dd272f3-4873-45e4-ac0d-2dfd9457fea5" (UID: "6dd272f3-4873-45e4-ac0d-2dfd9457fea5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:30.405146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.405117 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-util" pod "6dd272f3-4873-45e4-ac0d-2dfd9457fea5" (UID: "6dd272f3-4873-45e4-ac0d-2dfd9457fea5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:30.405747 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.405710 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-kube-api-access-rtdmx" pod "6dd272f3-4873-45e4-ac0d-2dfd9457fea5" (UID: "6dd272f3-4873-45e4-ac0d-2dfd9457fea5"). InnerVolumeSpecName "kube-api-access-rtdmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:30.483321 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.483238 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" podStartSLOduration=3.359688616 podStartE2EDuration="43.483224357s" totalImagesPullingTime="40.123535741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:30.481168837 +0000 UTC m=+600.460751172" watchObservedRunningTime="2026-11-01 00:33:30.483224357 +0000 UTC m=+600.462806691" Nov 01 00:33:30.496916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.496861 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdmx\" (UniqueName: \"kubernetes.io/projected/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-kube-api-access-rtdmx\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:30.496916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.496900 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:30.496916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:30.496914 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dd272f3-4873-45e4-ac0d-2dfd9457fea5-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:31.141160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.141064 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:33:31.142243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.142217 4910 generic.go:356] "Generic (PLEG): container finished" podID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerID="d0e9d80e1d6f996812e904eb5c0f3282149c1ffc6da83982311e4f84752fc20c" exitCode=1 Nov 01 00:33:31.142338 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.142320 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" event={"ID":"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c","Type":"ContainerDied","Data":"d0e9d80e1d6f996812e904eb5c0f3282149c1ffc6da83982311e4f84752fc20c"} Nov 01 00:33:31.145298 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.145270 4910 generic.go:356] "Generic (PLEG): container finished" podID="7acaa955-c1c3-4640-aea4-8b659298da11" containerID="652ce3ae065d95a0dcdd438f8d7556c3e36aba1896554e946a40d4724b787991" exitCode=0 Nov 01 00:33:31.145397 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.145352 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" event={"ID":"7acaa955-c1c3-4640-aea4-8b659298da11","Type":"ContainerDied","Data":"652ce3ae065d95a0dcdd438f8d7556c3e36aba1896554e946a40d4724b787991"} Nov 01 00:33:31.147904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.147877 4910 generic.go:356] "Generic (PLEG): container finished" podID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerID="56fafff0b6a7ac123b9ec9e396fa5c6c3062a52d11e1db6f6ae1f30842c6acc4" exitCode=0 Nov 01 00:33:31.148025 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.147990 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" event={"ID":"3bc446fe-d5bb-4080-9ae5-775a73321a58","Type":"ContainerDied","Data":"56fafff0b6a7ac123b9ec9e396fa5c6c3062a52d11e1db6f6ae1f30842c6acc4"} Nov 01 00:33:31.150551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.150527 4910 generic.go:356] "Generic (PLEG): container finished" podID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerID="92dc3240fd34c7ccff3808387af82fb8800a95c3ed2ef6a098fb3a6d30828c73" exitCode=0 Nov 01 00:33:31.150692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.150644 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" event={"ID":"28944ecf-df2d-44c8-9d6e-49c2a803f813","Type":"ContainerDied","Data":"92dc3240fd34c7ccff3808387af82fb8800a95c3ed2ef6a098fb3a6d30828c73"} Nov 01 00:33:31.152964 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.152941 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" event={"ID":"6dd272f3-4873-45e4-ac0d-2dfd9457fea5","Type":"ContainerDied","Data":"1eae03030752bcf6b0fdb6f0e47cae2555303dd0772d3a82973cd41defd76c77"} Nov 01 00:33:31.153049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.152983 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eae03030752bcf6b0fdb6f0e47cae2555303dd0772d3a82973cd41defd76c77" Nov 01 00:33:31.153128 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.153069 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm9dxk" Nov 01 00:33:31.164673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.164646 4910 generic.go:356] "Generic (PLEG): container finished" podID="c25f286c-8837-4637-be75-1e1c0f611042" containerID="56ae5af20f8e22b4b3c09bc76b6ce9b0a36c599ea0979e6ac5575495d4ef484e" exitCode=0 Nov 01 00:33:31.164764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.164717 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" event={"ID":"c25f286c-8837-4637-be75-1e1c0f611042","Type":"ContainerDied","Data":"56ae5af20f8e22b4b3c09bc76b6ce9b0a36c599ea0979e6ac5575495d4ef484e"} Nov 01 00:33:31.167393 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.167367 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" event={"ID":"ae231ee1-1de0-4df6-8b1d-0c2e249098f3","Type":"ContainerStarted","Data":"0869d90f02ced153b6ffcea331efc56328d59511d4bc1caddc315032e2843c88"} Nov 01 00:33:31.334222 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.334195 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:33:31.371432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.371401 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:33:31.406561 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.406528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef603bf-0e60-4882-bdfc-706a86af05ea-kube-api-access-l9lhf\" (UniqueName: \"kubernetes.io/projected/cef603bf-0e60-4882-bdfc-706a86af05ea-kube-api-access-l9lhf\") pod \"cef603bf-0e60-4882-bdfc-706a86af05ea\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " Nov 01 00:33:31.406774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.406579 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-bundle\") pod \"cef603bf-0e60-4882-bdfc-706a86af05ea\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " Nov 01 00:33:31.406774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.406692 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-util\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-util\") pod \"cef603bf-0e60-4882-bdfc-706a86af05ea\" (UID: \"cef603bf-0e60-4882-bdfc-706a86af05ea\") " Nov 01 00:33:31.407853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.407824 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-bundle" pod "cef603bf-0e60-4882-bdfc-706a86af05ea" (UID: "cef603bf-0e60-4882-bdfc-706a86af05ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:31.413747 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.413710 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-util" pod "cef603bf-0e60-4882-bdfc-706a86af05ea" (UID: "cef603bf-0e60-4882-bdfc-706a86af05ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:31.416088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.416058 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef603bf-0e60-4882-bdfc-706a86af05ea-kube-api-access-l9lhf" pod "cef603bf-0e60-4882-bdfc-706a86af05ea" (UID: "cef603bf-0e60-4882-bdfc-706a86af05ea"). InnerVolumeSpecName "kube-api-access-l9lhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:31.502352 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.502285 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" podStartSLOduration=3.392967689 podStartE2EDuration="51.50226876s" totalImagesPullingTime="48.109301071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:31.50173413 +0000 UTC m=+601.481316446" watchObservedRunningTime="2026-11-01 00:33:31.50226876 +0000 UTC m=+601.481851097" Nov 01 00:33:31.507836 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.507804 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-bundle\") pod \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " Nov 01 00:33:31.507992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.507865 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-util\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-util\") pod \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " Nov 01 00:33:31.507992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.507937 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-kube-api-access-w2zbb\" (UniqueName: \"kubernetes.io/projected/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-kube-api-access-w2zbb\") pod \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\" (UID: \"a6de775b-8d31-41d8-8c59-cf2d36d17b3f\") " Nov 01 00:33:31.508246 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.508222 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9lhf\" (UniqueName: \"kubernetes.io/projected/cef603bf-0e60-4882-bdfc-706a86af05ea-kube-api-access-l9lhf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:31.508355 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.508246 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:31.508355 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.508261 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cef603bf-0e60-4882-bdfc-706a86af05ea-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:31.508911 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.508871 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-bundle" pod "a6de775b-8d31-41d8-8c59-cf2d36d17b3f" (UID: "a6de775b-8d31-41d8-8c59-cf2d36d17b3f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:31.511058 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.511027 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-kube-api-access-w2zbb" pod "a6de775b-8d31-41d8-8c59-cf2d36d17b3f" (UID: "a6de775b-8d31-41d8-8c59-cf2d36d17b3f"). InnerVolumeSpecName "kube-api-access-w2zbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:31.514157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.514132 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-util" pod "a6de775b-8d31-41d8-8c59-cf2d36d17b3f" (UID: "a6de775b-8d31-41d8-8c59-cf2d36d17b3f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:31.608947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.608853 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zbb\" (UniqueName: \"kubernetes.io/projected/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-kube-api-access-w2zbb\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:31.608947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.608895 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:31.608947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:31.608912 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6de775b-8d31-41d8-8c59-cf2d36d17b3f-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.174514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.174479 4910 generic.go:356] "Generic (PLEG): container finished" podID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerID="0869d90f02ced153b6ffcea331efc56328d59511d4bc1caddc315032e2843c88" exitCode=0 Nov 01 00:33:32.174977 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.174555 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" event={"ID":"ae231ee1-1de0-4df6-8b1d-0c2e249098f3","Type":"ContainerDied","Data":"0869d90f02ced153b6ffcea331efc56328d59511d4bc1caddc315032e2843c88"} Nov 01 00:33:32.176681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.176659 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" Nov 01 00:33:32.176797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.176685 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8n5xt" event={"ID":"a6de775b-8d31-41d8-8c59-cf2d36d17b3f","Type":"ContainerDied","Data":"2eb51e39a0930ab3d5f8b8d6ef3a8963e94bd4b699c0d19bc2f049a6b64aaf98"} Nov 01 00:33:32.176797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.176750 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb51e39a0930ab3d5f8b8d6ef3a8963e94bd4b699c0d19bc2f049a6b64aaf98" Nov 01 00:33:32.178781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.178757 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" event={"ID":"cef603bf-0e60-4882-bdfc-706a86af05ea","Type":"ContainerDied","Data":"dba492f43fb8d2d6f623dbd58a5c44dc79a4485791fdb0aa3d9c84339505ea78"} Nov 01 00:33:32.178896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.178803 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dba492f43fb8d2d6f623dbd58a5c44dc79a4485791fdb0aa3d9c84339505ea78" Nov 01 00:33:32.178896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.178761 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jl6qc" Nov 01 00:33:32.181223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.181184 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" event={"ID":"868d4588-719a-49c2-8f30-d1d02663c0c5","Type":"ContainerStarted","Data":"0cc3601a82b8753f67f389a98cb491876ae1e4d535f0c80a140ccbc24c8b45e9"} Nov 01 00:33:32.183576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.183549 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" event={"ID":"d289c0f5-5d64-4638-8121-4ac3df4e36b0","Type":"ContainerStarted","Data":"fe3a98c854daba77ec0bb6dda3a9d6feb6a515b2fb452698ef0ef4d302e454ff"} Nov 01 00:33:32.350320 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.350293 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:33:32.412599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.412575 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:33:32.413475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.413443 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:33:32.415487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.415465 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:33:32.418773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.418751 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:33:32.420422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.420404 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-bundle\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-bundle\") pod \"28944ecf-df2d-44c8-9d6e-49c2a803f813\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " Nov 01 00:33:32.420522 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.420499 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-util\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-util\") pod \"28944ecf-df2d-44c8-9d6e-49c2a803f813\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " Nov 01 00:33:32.420576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.420560 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28944ecf-df2d-44c8-9d6e-49c2a803f813-kube-api-access-xkbvc\" (UniqueName: \"kubernetes.io/projected/28944ecf-df2d-44c8-9d6e-49c2a803f813-kube-api-access-xkbvc\") pod \"28944ecf-df2d-44c8-9d6e-49c2a803f813\" (UID: \"28944ecf-df2d-44c8-9d6e-49c2a803f813\") " Nov 01 00:33:32.421175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.421146 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-bundle" pod "28944ecf-df2d-44c8-9d6e-49c2a803f813" (UID: "28944ecf-df2d-44c8-9d6e-49c2a803f813"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.422171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.422155 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:33:32.422823 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.422805 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28944ecf-df2d-44c8-9d6e-49c2a803f813-kube-api-access-xkbvc" pod "28944ecf-df2d-44c8-9d6e-49c2a803f813" (UID: "28944ecf-df2d-44c8-9d6e-49c2a803f813"). InnerVolumeSpecName "kube-api-access-xkbvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:32.425481 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.425458 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-util" pod "28944ecf-df2d-44c8-9d6e-49c2a803f813" (UID: "28944ecf-df2d-44c8-9d6e-49c2a803f813"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.520714 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.520597 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" podStartSLOduration=3.847288476 podStartE2EDuration="51.520582976s" totalImagesPullingTime="47.6732945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:32.5201982 +0000 UTC m=+602.499780548" watchObservedRunningTime="2026-11-01 00:33:32.520582976 +0000 UTC m=+602.500165310" Nov 01 00:33:32.521220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521201 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-util\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-util\") pod \"3bc446fe-d5bb-4080-9ae5-775a73321a58\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " Nov 01 00:33:32.521268 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521245 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-util\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-util\") pod \"7acaa955-c1c3-4640-aea4-8b659298da11\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " Nov 01 00:33:32.521310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521281 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc446fe-d5bb-4080-9ae5-775a73321a58-kube-api-access-8wrv7\" (UniqueName: \"kubernetes.io/projected/3bc446fe-d5bb-4080-9ae5-775a73321a58-kube-api-access-8wrv7\") pod \"3bc446fe-d5bb-4080-9ae5-775a73321a58\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " Nov 01 00:33:32.521352 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521324 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-bundle\") pod \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " Nov 01 00:33:32.521399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521362 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-util\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-util\") pod \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " Nov 01 00:33:32.521399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521391 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-kube-api-access-v2mn8\" (UniqueName: \"kubernetes.io/projected/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-kube-api-access-v2mn8\") pod \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\" (UID: \"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c\") " Nov 01 00:33:32.521498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521433 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25f286c-8837-4637-be75-1e1c0f611042-kube-api-access-cpvfj\" (UniqueName: \"kubernetes.io/projected/c25f286c-8837-4637-be75-1e1c0f611042-kube-api-access-cpvfj\") pod \"c25f286c-8837-4637-be75-1e1c0f611042\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " Nov 01 00:33:32.521498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521467 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acaa955-c1c3-4640-aea4-8b659298da11-kube-api-access-k5472\" (UniqueName: \"kubernetes.io/projected/7acaa955-c1c3-4640-aea4-8b659298da11-kube-api-access-k5472\") pod \"7acaa955-c1c3-4640-aea4-8b659298da11\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " Nov 01 00:33:32.521606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521507 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-bundle\") pod \"c25f286c-8837-4637-be75-1e1c0f611042\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " Nov 01 00:33:32.521606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521547 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-bundle\") pod \"7acaa955-c1c3-4640-aea4-8b659298da11\" (UID: \"7acaa955-c1c3-4640-aea4-8b659298da11\") " Nov 01 00:33:32.521606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521573 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-util\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-util\") pod \"c25f286c-8837-4637-be75-1e1c0f611042\" (UID: \"c25f286c-8837-4637-be75-1e1c0f611042\") " Nov 01 00:33:32.521787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521657 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-bundle\") pod \"3bc446fe-d5bb-4080-9ae5-775a73321a58\" (UID: \"3bc446fe-d5bb-4080-9ae5-775a73321a58\") " Nov 01 00:33:32.522671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521911 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.522671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521929 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28944ecf-df2d-44c8-9d6e-49c2a803f813-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.522671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.521943 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbvc\" (UniqueName: \"kubernetes.io/projected/28944ecf-df2d-44c8-9d6e-49c2a803f813-kube-api-access-xkbvc\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.522671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.522014 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-bundle" pod "b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" (UID: "b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.522671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.522412 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-bundle" pod "c25f286c-8837-4637-be75-1e1c0f611042" (UID: "c25f286c-8837-4637-be75-1e1c0f611042"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.522671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.522442 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-bundle" pod "3bc446fe-d5bb-4080-9ae5-775a73321a58" (UID: "3bc446fe-d5bb-4080-9ae5-775a73321a58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.523297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.523259 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-bundle" pod "7acaa955-c1c3-4640-aea4-8b659298da11" (UID: "7acaa955-c1c3-4640-aea4-8b659298da11"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.524587 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.524566 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc446fe-d5bb-4080-9ae5-775a73321a58-kube-api-access-8wrv7" pod "3bc446fe-d5bb-4080-9ae5-775a73321a58" (UID: "3bc446fe-d5bb-4080-9ae5-775a73321a58"). InnerVolumeSpecName "kube-api-access-8wrv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:32.524926 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.524906 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25f286c-8837-4637-be75-1e1c0f611042-kube-api-access-cpvfj" pod "c25f286c-8837-4637-be75-1e1c0f611042" (UID: "c25f286c-8837-4637-be75-1e1c0f611042"). InnerVolumeSpecName "kube-api-access-cpvfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:32.525554 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.525531 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acaa955-c1c3-4640-aea4-8b659298da11-kube-api-access-k5472" pod "7acaa955-c1c3-4640-aea4-8b659298da11" (UID: "7acaa955-c1c3-4640-aea4-8b659298da11"). InnerVolumeSpecName "kube-api-access-k5472". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:32.526270 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.526251 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-kube-api-access-v2mn8" pod "b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" (UID: "b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c"). InnerVolumeSpecName "kube-api-access-v2mn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:32.527492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.527472 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-util" pod "b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" (UID: "b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.529533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.529513 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-util" pod "7acaa955-c1c3-4640-aea4-8b659298da11" (UID: "7acaa955-c1c3-4640-aea4-8b659298da11"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.530451 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.530426 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-util" pod "3bc446fe-d5bb-4080-9ae5-775a73321a58" (UID: "3bc446fe-d5bb-4080-9ae5-775a73321a58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.531483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.531462 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-util" pod "c25f286c-8837-4637-be75-1e1c0f611042" (UID: "c25f286c-8837-4637-be75-1e1c0f611042"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:32.622831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622796 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.622831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622825 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.622831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622834 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wrv7\" (UniqueName: \"kubernetes.io/projected/3bc446fe-d5bb-4080-9ae5-775a73321a58-kube-api-access-8wrv7\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622844 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622854 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622862 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2mn8\" (UniqueName: \"kubernetes.io/projected/b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c-kube-api-access-v2mn8\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622870 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvfj\" (UniqueName: \"kubernetes.io/projected/c25f286c-8837-4637-be75-1e1c0f611042-kube-api-access-cpvfj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622880 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5472\" (UniqueName: \"kubernetes.io/projected/7acaa955-c1c3-4640-aea4-8b659298da11-kube-api-access-k5472\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622890 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622898 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7acaa955-c1c3-4640-aea4-8b659298da11-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622905 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25f286c-8837-4637-be75-1e1c0f611042-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.623066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.622913 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc446fe-d5bb-4080-9ae5-775a73321a58-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:32.638987 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:32.638939 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" podStartSLOduration=3.26260826 podStartE2EDuration="48.638924311s" totalImagesPullingTime="45.376316051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:32:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:32.638050994 +0000 UTC m=+602.617633320" watchObservedRunningTime="2026-11-01 00:33:32.638924311 +0000 UTC m=+602.618506646" Nov 01 00:33:33.190160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.190120 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" event={"ID":"28944ecf-df2d-44c8-9d6e-49c2a803f813","Type":"ContainerDied","Data":"2fdc85834346fe3a1d17f3f29d038cba55868949c5d254404f77fd0267286534"} Nov 01 00:33:33.190160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.190156 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amwhxl" Nov 01 00:33:33.190723 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.190167 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fdc85834346fe3a1d17f3f29d038cba55868949c5d254404f77fd0267286534" Nov 01 00:33:33.192426 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.192340 4910 generic.go:356] "Generic (PLEG): container finished" podID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerID="0cc3601a82b8753f67f389a98cb491876ae1e4d535f0c80a140ccbc24c8b45e9" exitCode=0 Nov 01 00:33:33.192555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.192438 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" event={"ID":"868d4588-719a-49c2-8f30-d1d02663c0c5","Type":"ContainerDied","Data":"0cc3601a82b8753f67f389a98cb491876ae1e4d535f0c80a140ccbc24c8b45e9"} Nov 01 00:33:33.194494 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.194471 4910 generic.go:356] "Generic (PLEG): container finished" podID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerID="fe3a98c854daba77ec0bb6dda3a9d6feb6a515b2fb452698ef0ef4d302e454ff" exitCode=0 Nov 01 00:33:33.194614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.194557 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" event={"ID":"d289c0f5-5d64-4638-8121-4ac3df4e36b0","Type":"ContainerDied","Data":"fe3a98c854daba77ec0bb6dda3a9d6feb6a515b2fb452698ef0ef4d302e454ff"} Nov 01 00:33:33.196373 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.196357 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" Nov 01 00:33:33.196475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.196359 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69bxqn" event={"ID":"c25f286c-8837-4637-be75-1e1c0f611042","Type":"ContainerDied","Data":"0f0ec715ba575bbea24a55258319b756a2036100bd7f711ebf95e2843210b48f"} Nov 01 00:33:33.196523 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.196482 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0ec715ba575bbea24a55258319b756a2036100bd7f711ebf95e2843210b48f" Nov 01 00:33:33.197668 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.197653 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:33:33.198333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.198312 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" event={"ID":"b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c","Type":"ContainerDied","Data":"5b855fd2be6cba5d69ef8d0f525a4d124c537e2ebfb73d4cb80cf31042ee5dcf"} Nov 01 00:33:33.198333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.198324 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg" Nov 01 00:33:33.198493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.198347 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b855fd2be6cba5d69ef8d0f525a4d124c537e2ebfb73d4cb80cf31042ee5dcf" Nov 01 00:33:33.200641 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.200588 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" event={"ID":"7acaa955-c1c3-4640-aea4-8b659298da11","Type":"ContainerDied","Data":"0fe7073727bf1941aa0620464d6a1ca49e4aa120d4dbeb66eb87e0d44ab4d6a8"} Nov 01 00:33:33.200735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.200649 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe7073727bf1941aa0620464d6a1ca49e4aa120d4dbeb66eb87e0d44ab4d6a8" Nov 01 00:33:33.200735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.200701 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajhkq6" Nov 01 00:33:33.202486 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.202470 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" Nov 01 00:33:33.202575 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.202499 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rj8n2" event={"ID":"3bc446fe-d5bb-4080-9ae5-775a73321a58","Type":"ContainerDied","Data":"0e32b5270e45d413b3450d6ffb316a50eb78b33e7fe155b00b31d5eb5170ea47"} Nov 01 00:33:33.202575 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.202533 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e32b5270e45d413b3450d6ffb316a50eb78b33e7fe155b00b31d5eb5170ea47" Nov 01 00:33:33.339694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.339667 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:33:33.430926 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.430842 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-kube-api-access-95xsg\" (UniqueName: \"kubernetes.io/projected/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-kube-api-access-95xsg\") pod \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " Nov 01 00:33:33.431078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.430936 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-bundle\") pod \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " Nov 01 00:33:33.431078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.430970 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-util\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-util\") pod \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\" (UID: \"ae231ee1-1de0-4df6-8b1d-0c2e249098f3\") " Nov 01 00:33:33.431407 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.431387 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-bundle" pod "ae231ee1-1de0-4df6-8b1d-0c2e249098f3" (UID: "ae231ee1-1de0-4df6-8b1d-0c2e249098f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:33.433176 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.433145 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-kube-api-access-95xsg" pod "ae231ee1-1de0-4df6-8b1d-0c2e249098f3" (UID: "ae231ee1-1de0-4df6-8b1d-0c2e249098f3"). InnerVolumeSpecName "kube-api-access-95xsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:33.436688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.436663 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-util" pod "ae231ee1-1de0-4df6-8b1d-0c2e249098f3" (UID: "ae231ee1-1de0-4df6-8b1d-0c2e249098f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:33.531746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.531710 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xsg\" (UniqueName: \"kubernetes.io/projected/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-kube-api-access-95xsg\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:33.531746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.531741 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:33.531746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:33.531753 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae231ee1-1de0-4df6-8b1d-0c2e249098f3-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:34.208893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.208870 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" Nov 01 00:33:34.209333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.208903 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tjjv2" event={"ID":"ae231ee1-1de0-4df6-8b1d-0c2e249098f3","Type":"ContainerDied","Data":"860caef6cf5660917c19437e8bb6e019d22f97508fc672e144218b16bef0b3ef"} Nov 01 00:33:34.209333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.208949 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="860caef6cf5660917c19437e8bb6e019d22f97508fc672e144218b16bef0b3ef" Nov 01 00:33:34.358717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.358681 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:33:34.361279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.361257 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:33:34.440027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.439999 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-util\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-util\") pod \"868d4588-719a-49c2-8f30-d1d02663c0c5\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " Nov 01 00:33:34.440027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.440031 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-bundle\") pod \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " Nov 01 00:33:34.440237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.440050 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d289c0f5-5d64-4638-8121-4ac3df4e36b0-kube-api-access-zd7f7\" (UniqueName: \"kubernetes.io/projected/d289c0f5-5d64-4638-8121-4ac3df4e36b0-kube-api-access-zd7f7\") pod \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " Nov 01 00:33:34.440237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.440085 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868d4588-719a-49c2-8f30-d1d02663c0c5-kube-api-access-59kb7\" (UniqueName: \"kubernetes.io/projected/868d4588-719a-49c2-8f30-d1d02663c0c5-kube-api-access-59kb7\") pod \"868d4588-719a-49c2-8f30-d1d02663c0c5\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " Nov 01 00:33:34.440237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.440154 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-util\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-util\") pod \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\" (UID: \"d289c0f5-5d64-4638-8121-4ac3df4e36b0\") " Nov 01 00:33:34.440237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.440204 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-bundle\") pod \"868d4588-719a-49c2-8f30-d1d02663c0c5\" (UID: \"868d4588-719a-49c2-8f30-d1d02663c0c5\") " Nov 01 00:33:34.440722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.440695 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-bundle" pod "868d4588-719a-49c2-8f30-d1d02663c0c5" (UID: "868d4588-719a-49c2-8f30-d1d02663c0c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:34.440962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.440886 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-bundle" pod "d289c0f5-5d64-4638-8121-4ac3df4e36b0" (UID: "d289c0f5-5d64-4638-8121-4ac3df4e36b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:34.442363 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.442343 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d289c0f5-5d64-4638-8121-4ac3df4e36b0-kube-api-access-zd7f7" pod "d289c0f5-5d64-4638-8121-4ac3df4e36b0" (UID: "d289c0f5-5d64-4638-8121-4ac3df4e36b0"). InnerVolumeSpecName "kube-api-access-zd7f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:34.442472 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.442454 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868d4588-719a-49c2-8f30-d1d02663c0c5-kube-api-access-59kb7" pod "868d4588-719a-49c2-8f30-d1d02663c0c5" (UID: "868d4588-719a-49c2-8f30-d1d02663c0c5"). InnerVolumeSpecName "kube-api-access-59kb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:33:34.445985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.445963 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-util" pod "d289c0f5-5d64-4638-8121-4ac3df4e36b0" (UID: "d289c0f5-5d64-4638-8121-4ac3df4e36b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:34.447401 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.447376 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-util" pod "868d4588-719a-49c2-8f30-d1d02663c0c5" (UID: "868d4588-719a-49c2-8f30-d1d02663c0c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:33:34.541704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.541670 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:34.541704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.541693 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:34.541704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.541701 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/868d4588-719a-49c2-8f30-d1d02663c0c5-util\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:34.541704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.541709 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d289c0f5-5d64-4638-8121-4ac3df4e36b0-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:34.541704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.541717 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7f7\" (UniqueName: \"kubernetes.io/projected/d289c0f5-5d64-4638-8121-4ac3df4e36b0-kube-api-access-zd7f7\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:34.542116 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.541732 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59kb7\" (UniqueName: \"kubernetes.io/projected/868d4588-719a-49c2-8f30-d1d02663c0c5-kube-api-access-59kb7\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:33:34.791946 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.791911 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx"] Nov 01 00:33:34.792241 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792229 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerName="pull" Nov 01 00:33:34.792281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792243 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerName="pull" Nov 01 00:33:34.792281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792252 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerName="extract" Nov 01 00:33:34.792281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792257 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerName="extract" Nov 01 00:33:34.792281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792263 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c25f286c-8837-4637-be75-1e1c0f611042" containerName="extract" Nov 01 00:33:34.792281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792268 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25f286c-8837-4637-be75-1e1c0f611042" containerName="extract" Nov 01 00:33:34.792281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792276 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerName="pull" Nov 01 00:33:34.792281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792280 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792288 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792292 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792299 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7acaa955-c1c3-4640-aea4-8b659298da11" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792303 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acaa955-c1c3-4640-aea4-8b659298da11" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792309 4910 cpu_manager.go:414] "containerMap: removing container" podUID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792314 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792319 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792323 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792329 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792333 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792340 4910 cpu_manager.go:414] "containerMap: removing container" podUID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792344 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792350 4910 cpu_manager.go:414] "containerMap: removing container" podUID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792354 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792359 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792363 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792369 4910 cpu_manager.go:414] "containerMap: removing container" podUID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792373 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792383 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792388 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792398 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792402 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792410 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792414 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792420 4910 cpu_manager.go:414] "containerMap: removing container" podUID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792425 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792431 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7acaa955-c1c3-4640-aea4-8b659298da11" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792436 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acaa955-c1c3-4640-aea4-8b659298da11" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792441 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792446 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792453 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792457 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792465 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792470 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerName="extract" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792477 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792481 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792487 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792491 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerName="pull" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792498 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerName="util" Nov 01 00:33:34.792484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792502 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792509 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c25f286c-8837-4637-be75-1e1c0f611042" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792513 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25f286c-8837-4637-be75-1e1c0f611042" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792519 4910 cpu_manager.go:414] "containerMap: removing container" podUID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792523 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792528 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792533 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792539 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c25f286c-8837-4637-be75-1e1c0f611042" containerName="pull" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792543 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25f286c-8837-4637-be75-1e1c0f611042" containerName="pull" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792548 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7acaa955-c1c3-4640-aea4-8b659298da11" containerName="pull" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792552 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acaa955-c1c3-4640-aea4-8b659298da11" containerName="pull" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792558 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792562 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792567 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792571 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792581 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792584 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerName="util" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792590 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerName="pull" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792594 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerName="pull" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792660 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae231ee1-1de0-4df6-8b1d-0c2e249098f3" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792669 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="28944ecf-df2d-44c8-9d6e-49c2a803f813" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792682 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd272f3-4873-45e4-ac0d-2dfd9457fea5" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792693 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792698 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6de775b-8d31-41d8-8c59-cf2d36d17b3f" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792704 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef603bf-0e60-4882-bdfc-706a86af05ea" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792709 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d289c0f5-5d64-4638-8121-4ac3df4e36b0" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792713 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25f286c-8837-4637-be75-1e1c0f611042" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792719 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc446fe-d5bb-4080-9ae5-775a73321a58" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792724 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acaa955-c1c3-4640-aea4-8b659298da11" containerName="extract" Nov 01 00:33:34.793637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.792729 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="868d4588-719a-49c2-8f30-d1d02663c0c5" containerName="extract" Nov 01 00:33:34.794805 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.794789 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:34.849544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.849512 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 00:33:34.849814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.849795 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 00:33:34.849909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.849892 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 00:33:34.849966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.849930 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 00:33:34.850048 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.850036 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 00:33:34.850257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.850241 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-28jl7\"" Nov 01 00:33:34.928512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.928479 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx"] Nov 01 00:33:34.945764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.945732 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskkj\" (UniqueName: \"kubernetes.io/projected/a591148d-9810-41f2-9f7e-f7526240ac22-kube-api-access-bskkj\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:34.945924 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.945777 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:34.945924 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.945869 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:34.946013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.945960 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:34.946013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.945979 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a591148d-9810-41f2-9f7e-f7526240ac22-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:34.946013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:34.946004 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a591148d-9810-41f2-9f7e-f7526240ac22-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.046812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.046736 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.046812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.046769 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a591148d-9810-41f2-9f7e-f7526240ac22-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.046812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.046794 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a591148d-9810-41f2-9f7e-f7526240ac22-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.047107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.046918 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskkj\" (UniqueName: \"kubernetes.io/projected/a591148d-9810-41f2-9f7e-f7526240ac22-kube-api-access-bskkj\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.047107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.046959 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.047107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.046982 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.047257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.047129 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a591148d-9810-41f2-9f7e-f7526240ac22-manifests\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.049613 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.049579 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.049774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.049579 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.049893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.049873 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a591148d-9810-41f2-9f7e-f7526240ac22-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.050099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.050078 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a591148d-9810-41f2-9f7e-f7526240ac22-tmp\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.108666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.108618 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskkj\" (UniqueName: \"kubernetes.io/projected/a591148d-9810-41f2-9f7e-f7526240ac22-kube-api-access-bskkj\") pod \"opendatahub-operator-controller-manager-7b76c875b9-5g8xx\" (UID: \"a591148d-9810-41f2-9f7e-f7526240ac22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.215041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.215007 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" event={"ID":"868d4588-719a-49c2-8f30-d1d02663c0c5","Type":"ContainerDied","Data":"d94682c0ed5ee8ba166ad2332f22487ce5492a66314f7ce4e43d90c2c56ceddf"} Nov 01 00:33:35.215041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.215043 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479ql6" Nov 01 00:33:35.215578 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.215051 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d94682c0ed5ee8ba166ad2332f22487ce5492a66314f7ce4e43d90c2c56ceddf" Nov 01 00:33:35.217050 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.217021 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" Nov 01 00:33:35.217180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.217046 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgc4kt" event={"ID":"d289c0f5-5d64-4638-8121-4ac3df4e36b0","Type":"ContainerDied","Data":"626faeac2c53e4fb66f7f7d2dc13e6c429bf6e64820eeb42db1e4807bce5e482"} Nov 01 00:33:35.217180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.217091 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="626faeac2c53e4fb66f7f7d2dc13e6c429bf6e64820eeb42db1e4807bce5e482" Nov 01 00:33:35.407027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.406938 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:35.963225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:35.963204 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx"] Nov 01 00:33:35.965124 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:33:35.965096 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda591148d_9810_41f2_9f7e_f7526240ac22.slice/crio-062c40c547b77ebe9d131203bcae6376bc1b24a8e40c50b7ae5d8ebba5e3fb5b WatchSource:0}: Error finding container 062c40c547b77ebe9d131203bcae6376bc1b24a8e40c50b7ae5d8ebba5e3fb5b: Status 404 returned error can't find the container with id 062c40c547b77ebe9d131203bcae6376bc1b24a8e40c50b7ae5d8ebba5e3fb5b Nov 01 00:33:36.222833 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:36.222751 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" event={"ID":"a591148d-9810-41f2-9f7e-f7526240ac22","Type":"ContainerStarted","Data":"062c40c547b77ebe9d131203bcae6376bc1b24a8e40c50b7ae5d8ebba5e3fb5b"} Nov 01 00:33:40.245037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:40.245004 4910 generic.go:356] "Generic (PLEG): container finished" podID="a591148d-9810-41f2-9f7e-f7526240ac22" containerID="33310b8eb5c2bd684aa4971fcd8faac2aff0776a7f5e02776e7d5ef4e772097b" exitCode=0 Nov 01 00:33:40.245425 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:40.245058 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" event={"ID":"a591148d-9810-41f2-9f7e-f7526240ac22","Type":"ContainerDied","Data":"33310b8eb5c2bd684aa4971fcd8faac2aff0776a7f5e02776e7d5ef4e772097b"} Nov 01 00:33:41.251873 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:41.251836 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" event={"ID":"a591148d-9810-41f2-9f7e-f7526240ac22","Type":"ContainerStarted","Data":"d456f31149526fee60df8ae96ca0f5e85797cf397e6d0feb7277d066a5c3aac6"} Nov 01 00:33:41.252261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:41.251917 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:41.490498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:41.490380 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" podStartSLOduration=2.630253016 podStartE2EDuration="7.490363351s" totalImagesPullingTime="4.365368807s" totalInitContainerRuntime="494.741528ms" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:41.439712235 +0000 UTC m=+611.419294572" watchObservedRunningTime="2026-11-01 00:33:41.490363351 +0000 UTC m=+611.469945686" Nov 01 00:33:42.884545 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:42.884510 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h"] Nov 01 00:33:42.886712 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:42.886696 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:42.996920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:42.996843 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 00:33:42.996920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:42.996882 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 00:33:42.996920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:42.996916 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4rfbd\"" Nov 01 00:33:42.999075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:42.999049 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h"] Nov 01 00:33:43.020275 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.020243 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhl7s\" (UniqueName: \"kubernetes.io/projected/c5155ad3-795d-4fc1-a84d-c86e8c76b39a-kube-api-access-hhl7s\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h\" (UID: \"c5155ad3-795d-4fc1-a84d-c86e8c76b39a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:43.020404 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.020329 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5155ad3-795d-4fc1-a84d-c86e8c76b39a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h\" (UID: \"c5155ad3-795d-4fc1-a84d-c86e8c76b39a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:43.101690 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:33:43.101655 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:33:43.121176 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.121143 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5155ad3-795d-4fc1-a84d-c86e8c76b39a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h\" (UID: \"c5155ad3-795d-4fc1-a84d-c86e8c76b39a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:43.121325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.121222 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhl7s\" (UniqueName: \"kubernetes.io/projected/c5155ad3-795d-4fc1-a84d-c86e8c76b39a-kube-api-access-hhl7s\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h\" (UID: \"c5155ad3-795d-4fc1-a84d-c86e8c76b39a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:43.121576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.121553 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5155ad3-795d-4fc1-a84d-c86e8c76b39a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h\" (UID: \"c5155ad3-795d-4fc1-a84d-c86e8c76b39a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:43.182636 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.182559 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhl7s\" (UniqueName: \"kubernetes.io/projected/c5155ad3-795d-4fc1-a84d-c86e8c76b39a-kube-api-access-hhl7s\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h\" (UID: \"c5155ad3-795d-4fc1-a84d-c86e8c76b39a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:43.198738 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.198709 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" Nov 01 00:33:43.687318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:43.687287 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h"] Nov 01 00:33:43.689256 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:33:43.689227 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5155ad3_795d_4fc1_a84d_c86e8c76b39a.slice/crio-3266aff7251b356048d9f66b910994e7d59a9c3be7ef2a1f48d3167b9018472c WatchSource:0}: Error finding container 3266aff7251b356048d9f66b910994e7d59a9c3be7ef2a1f48d3167b9018472c: Status 404 returned error can't find the container with id 3266aff7251b356048d9f66b910994e7d59a9c3be7ef2a1f48d3167b9018472c Nov 01 00:33:44.269667 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:44.269615 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" event={"ID":"c5155ad3-795d-4fc1-a84d-c86e8c76b39a","Type":"ContainerStarted","Data":"3266aff7251b356048d9f66b910994e7d59a9c3be7ef2a1f48d3167b9018472c"} Nov 01 00:33:47.287532 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:47.287500 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" event={"ID":"c5155ad3-795d-4fc1-a84d-c86e8c76b39a","Type":"ContainerStarted","Data":"2b949c0a7b390202c7d02099035d93907dbc7293662e4ee400b64a01f3374c53"} Nov 01 00:33:47.509084 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:47.509031 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-p2k9h" podStartSLOduration=2.250437996 podStartE2EDuration="5.509017735s" totalImagesPullingTime="3.258579739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:47.457674866 +0000 UTC m=+617.437257201" watchObservedRunningTime="2026-11-01 00:33:47.509017735 +0000 UTC m=+617.488600069" Nov 01 00:33:49.729602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.729571 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws"] Nov 01 00:33:49.733045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.733023 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:49.837180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.837151 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 00:33:49.837328 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.837249 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 00:33:49.837369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.837346 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vsw7v\"" Nov 01 00:33:49.839160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.839140 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws"] Nov 01 00:33:49.882299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.882222 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brb8\" (UniqueName: \"kubernetes.io/projected/7990857a-39eb-4667-9e4d-7740dbfd4a72-kube-api-access-2brb8\") pod \"cert-manager-webhook-5dbbbf75cc-6bxws\" (UID: \"7990857a-39eb-4667-9e4d-7740dbfd4a72\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:49.882299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.882277 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7990857a-39eb-4667-9e4d-7740dbfd4a72-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6bxws\" (UID: \"7990857a-39eb-4667-9e4d-7740dbfd4a72\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:49.983812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.983774 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brb8\" (UniqueName: \"kubernetes.io/projected/7990857a-39eb-4667-9e4d-7740dbfd4a72-kube-api-access-2brb8\") pod \"cert-manager-webhook-5dbbbf75cc-6bxws\" (UID: \"7990857a-39eb-4667-9e4d-7740dbfd4a72\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:49.983981 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:49.983825 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7990857a-39eb-4667-9e4d-7740dbfd4a72-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6bxws\" (UID: \"7990857a-39eb-4667-9e4d-7740dbfd4a72\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:50.049219 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:50.049191 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7990857a-39eb-4667-9e4d-7740dbfd4a72-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6bxws\" (UID: \"7990857a-39eb-4667-9e4d-7740dbfd4a72\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:50.049367 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:50.049282 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brb8\" (UniqueName: \"kubernetes.io/projected/7990857a-39eb-4667-9e4d-7740dbfd4a72-kube-api-access-2brb8\") pod \"cert-manager-webhook-5dbbbf75cc-6bxws\" (UID: \"7990857a-39eb-4667-9e4d-7740dbfd4a72\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:50.057711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:50.057690 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:50.430589 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:50.430566 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws"] Nov 01 00:33:50.432274 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:33:50.432248 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7990857a_39eb_4667_9e4d_7740dbfd4a72.slice/crio-7a286cdc43d5886bd16ab18f3dfaf41e8d7abc0c72d929e993ef1fe033cce2bf WatchSource:0}: Error finding container 7a286cdc43d5886bd16ab18f3dfaf41e8d7abc0c72d929e993ef1fe033cce2bf: Status 404 returned error can't find the container with id 7a286cdc43d5886bd16ab18f3dfaf41e8d7abc0c72d929e993ef1fe033cce2bf Nov 01 00:33:51.308163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:51.308120 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" event={"ID":"7990857a-39eb-4667-9e4d-7740dbfd4a72","Type":"ContainerStarted","Data":"7a286cdc43d5886bd16ab18f3dfaf41e8d7abc0c72d929e993ef1fe033cce2bf"} Nov 01 00:33:52.259352 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:52.259321 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b76c875b9-5g8xx" Nov 01 00:33:54.327336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:54.327298 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" event={"ID":"7990857a-39eb-4667-9e4d-7740dbfd4a72","Type":"ContainerStarted","Data":"eb363f2acaa5b2e22df983be27607b03b6edad1cf7d544113cb7390f160ddcc4"} Nov 01 00:33:54.327822 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:54.327426 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:33:54.448948 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:54.448837 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" podStartSLOduration=1.718788466 podStartE2EDuration="5.448818885s" totalImagesPullingTime="3.730030419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:33:54.44791551 +0000 UTC m=+624.427497849" watchObservedRunningTime="2026-11-01 00:33:54.448818885 +0000 UTC m=+624.428401221" Nov 01 00:33:56.037510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.037476 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt"] Nov 01 00:33:56.040946 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.040925 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" Nov 01 00:33:56.100767 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.100731 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 00:33:56.141547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.141519 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kllr\" (UniqueName: \"kubernetes.io/projected/aa8d2336-53a8-4548-ab8a-05774f06ee89-kube-api-access-6kllr\") pod \"authorino-operator-7b7bb4bc77-fp6nt\" (UID: \"aa8d2336-53a8-4548-ab8a-05774f06ee89\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" Nov 01 00:33:56.150785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.150762 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 00:33:56.150785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.150779 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8rmxg\"" Nov 01 00:33:56.152988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.152968 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt"] Nov 01 00:33:56.242870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.242837 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kllr\" (UniqueName: \"kubernetes.io/projected/aa8d2336-53a8-4548-ab8a-05774f06ee89-kube-api-access-6kllr\") pod \"authorino-operator-7b7bb4bc77-fp6nt\" (UID: \"aa8d2336-53a8-4548-ab8a-05774f06ee89\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" Nov 01 00:33:56.303913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.303844 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kllr\" (UniqueName: \"kubernetes.io/projected/aa8d2336-53a8-4548-ab8a-05774f06ee89-kube-api-access-6kllr\") pod \"authorino-operator-7b7bb4bc77-fp6nt\" (UID: \"aa8d2336-53a8-4548-ab8a-05774f06ee89\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" Nov 01 00:33:56.353142 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.353114 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" Nov 01 00:33:56.729250 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:56.729224 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt"] Nov 01 00:33:56.730875 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:33:56.730847 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8d2336_53a8_4548_ab8a_05774f06ee89.slice/crio-7cd145e34e7a8689099112dbf1946e665f107767897369af084d0af04ad5043a WatchSource:0}: Error finding container 7cd145e34e7a8689099112dbf1946e665f107767897369af084d0af04ad5043a: Status 404 returned error can't find the container with id 7cd145e34e7a8689099112dbf1946e665f107767897369af084d0af04ad5043a Nov 01 00:33:57.342718 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:57.342685 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" event={"ID":"aa8d2336-53a8-4548-ab8a-05774f06ee89","Type":"ContainerStarted","Data":"7cd145e34e7a8689099112dbf1946e665f107767897369af084d0af04ad5043a"} Nov 01 00:33:59.835507 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:59.835472 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw"] Nov 01 00:33:59.838856 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:59.838840 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" Nov 01 00:33:59.872518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:59.872487 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvx4\" (UniqueName: \"kubernetes.io/projected/bbd2d693-1efb-4fd4-b23d-a9c4592ff185-kube-api-access-wkvx4\") pod \"limitador-operator-controller-manager-5778bb4449-vdplw\" (UID: \"bbd2d693-1efb-4fd4-b23d-a9c4592ff185\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" Nov 01 00:33:59.947034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:59.947002 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zrmd2\"" Nov 01 00:33:59.949298 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:59.949267 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw"] Nov 01 00:33:59.973914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:33:59.973876 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvx4\" (UniqueName: \"kubernetes.io/projected/bbd2d693-1efb-4fd4-b23d-a9c4592ff185-kube-api-access-wkvx4\") pod \"limitador-operator-controller-manager-5778bb4449-vdplw\" (UID: \"bbd2d693-1efb-4fd4-b23d-a9c4592ff185\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" Nov 01 00:34:00.045200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:00.045166 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvx4\" (UniqueName: \"kubernetes.io/projected/bbd2d693-1efb-4fd4-b23d-a9c4592ff185-kube-api-access-wkvx4\") pod \"limitador-operator-controller-manager-5778bb4449-vdplw\" (UID: \"bbd2d693-1efb-4fd4-b23d-a9c4592ff185\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" Nov 01 00:34:00.151425 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:00.151344 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" Nov 01 00:34:00.335724 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:00.335694 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6bxws" Nov 01 00:34:00.620301 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:00.620271 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw"] Nov 01 00:34:00.623010 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:34:00.622974 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd2d693_1efb_4fd4_b23d_a9c4592ff185.slice/crio-c16a071227699a54d85483a8f387abd548f826ea20f8c57d93e4670e8e1619de WatchSource:0}: Error finding container c16a071227699a54d85483a8f387abd548f826ea20f8c57d93e4670e8e1619de: Status 404 returned error can't find the container with id c16a071227699a54d85483a8f387abd548f826ea20f8c57d93e4670e8e1619de Nov 01 00:34:01.370348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:01.370305 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" event={"ID":"aa8d2336-53a8-4548-ab8a-05774f06ee89","Type":"ContainerStarted","Data":"cd0e7db43bb4790f9832da65369b6505eeffdd21adc57ae54acab4f19c61bce4"} Nov 01 00:34:01.370837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:01.370380 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" Nov 01 00:34:01.371725 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:01.371701 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" event={"ID":"bbd2d693-1efb-4fd4-b23d-a9c4592ff185","Type":"ContainerStarted","Data":"c16a071227699a54d85483a8f387abd548f826ea20f8c57d93e4670e8e1619de"} Nov 01 00:34:01.508889 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:01.508486 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" podStartSLOduration=2.021408777 podStartE2EDuration="6.508467404s" totalImagesPullingTime="4.487058627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:01.507836226 +0000 UTC m=+631.487418563" watchObservedRunningTime="2026-11-01 00:34:01.508467404 +0000 UTC m=+631.488049741" Nov 01 00:34:03.384107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:03.384072 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" event={"ID":"bbd2d693-1efb-4fd4-b23d-a9c4592ff185","Type":"ContainerStarted","Data":"924e010cbc67083f4fb70a4b26cdafdad90d54f808e979ade06a43a5e0c534d0"} Nov 01 00:34:03.384567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:03.384136 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" Nov 01 00:34:03.501606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:03.501541 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" podStartSLOduration=1.9497393010000001 podStartE2EDuration="4.501526733s" totalImagesPullingTime="2.551787432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:03.500195354 +0000 UTC m=+633.479777692" watchObservedRunningTime="2026-11-01 00:34:03.501526733 +0000 UTC m=+633.481109064" Nov 01 00:34:06.253560 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.253527 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q"] Nov 01 00:34:06.257246 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.257229 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:06.307532 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.307491 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q"] Nov 01 00:34:06.330240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.330198 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ncx6q\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:06.330388 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.330364 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrt9d\" (UniqueName: \"kubernetes.io/projected/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-kube-api-access-mrt9d\") pod \"kuadrant-operator-controller-manager-678d87766f-ncx6q\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:06.354752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.354716 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-crws2\"" Nov 01 00:34:06.431281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.431250 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrt9d\" (UniqueName: \"kubernetes.io/projected/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-kube-api-access-mrt9d\") pod \"kuadrant-operator-controller-manager-678d87766f-ncx6q\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:06.431454 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.431288 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ncx6q\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:06.431739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.431722 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ncx6q\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:06.492559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.492527 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrt9d\" (UniqueName: \"kubernetes.io/projected/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-kube-api-access-mrt9d\") pod \"kuadrant-operator-controller-manager-678d87766f-ncx6q\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:06.569835 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:06.569747 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:07.046020 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:07.045999 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q"] Nov 01 00:34:07.047994 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:34:07.047970 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c9dd7_b9a9_4a37_b6b0_dd726d030744.slice/crio-ca1fc937da0d1c56a40f83f4fcc4562dce7b4d6f0535155c764acdcd954a2799 WatchSource:0}: Error finding container ca1fc937da0d1c56a40f83f4fcc4562dce7b4d6f0535155c764acdcd954a2799: Status 404 returned error can't find the container with id ca1fc937da0d1c56a40f83f4fcc4562dce7b4d6f0535155c764acdcd954a2799 Nov 01 00:34:07.405751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:07.405653 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" event={"ID":"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744","Type":"ContainerStarted","Data":"ca1fc937da0d1c56a40f83f4fcc4562dce7b4d6f0535155c764acdcd954a2799"} Nov 01 00:34:08.426778 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.426744 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx"] Nov 01 00:34:08.431325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.431309 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" Nov 01 00:34:08.536152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.536121 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 00:34:08.536152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.536127 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f78ds\"" Nov 01 00:34:08.538252 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.538229 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx"] Nov 01 00:34:08.549493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.549473 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcs5\" (UniqueName: \"kubernetes.io/projected/4cda9f88-23b8-4d10-98b9-de79a789d559-kube-api-access-fdcs5\") pod \"dns-operator-controller-manager-8557cbdb84-tx8vx\" (UID: \"4cda9f88-23b8-4d10-98b9-de79a789d559\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" Nov 01 00:34:08.650942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.650914 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcs5\" (UniqueName: \"kubernetes.io/projected/4cda9f88-23b8-4d10-98b9-de79a789d559-kube-api-access-fdcs5\") pod \"dns-operator-controller-manager-8557cbdb84-tx8vx\" (UID: \"4cda9f88-23b8-4d10-98b9-de79a789d559\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" Nov 01 00:34:08.712259 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.712186 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcs5\" (UniqueName: \"kubernetes.io/projected/4cda9f88-23b8-4d10-98b9-de79a789d559-kube-api-access-fdcs5\") pod \"dns-operator-controller-manager-8557cbdb84-tx8vx\" (UID: \"4cda9f88-23b8-4d10-98b9-de79a789d559\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" Nov 01 00:34:08.745009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:08.744955 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" Nov 01 00:34:09.118572 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:09.118528 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx"] Nov 01 00:34:09.120914 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:34:09.120890 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cda9f88_23b8_4d10_98b9_de79a789d559.slice/crio-e67d9a2234cff1392b278126cd12b57e61cef452467982cff4863557f8f4b538 WatchSource:0}: Error finding container e67d9a2234cff1392b278126cd12b57e61cef452467982cff4863557f8f4b538: Status 404 returned error can't find the container with id e67d9a2234cff1392b278126cd12b57e61cef452467982cff4863557f8f4b538 Nov 01 00:34:09.417213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:09.417128 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" event={"ID":"4cda9f88-23b8-4d10-98b9-de79a789d559","Type":"ContainerStarted","Data":"e67d9a2234cff1392b278126cd12b57e61cef452467982cff4863557f8f4b538"} Nov 01 00:34:12.378707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:12.378676 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fp6nt" Nov 01 00:34:13.106465 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:34:13.106422 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:34:14.373874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.373839 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd"] Nov 01 00:34:14.377580 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.377558 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.390950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.390932 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vdplw" Nov 01 00:34:14.457881 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.457844 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd"] Nov 01 00:34:14.500470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.500439 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rlcf2\"" Nov 01 00:34:14.501780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.501755 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.501901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.501814 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mk28\" (UniqueName: \"kubernetes.io/projected/4617ca5d-ea03-492c-a926-924ddf3b5860-kube-api-access-6mk28\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.501965 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.501943 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.502011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.501995 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4617ca5d-ea03-492c-a926-924ddf3b5860-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.502134 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.502114 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.502186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.502143 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.502186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.502162 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.502331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.502312 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.502406 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.502337 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.603679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.603642 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.603679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.603681 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.603910 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.603836 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.603910 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.603891 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mk28\" (UniqueName: \"kubernetes.io/projected/4617ca5d-ea03-492c-a926-924ddf3b5860-kube-api-access-6mk28\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.603937 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.603969 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4617ca5d-ea03-492c-a926-924ddf3b5860-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604008 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604056 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604066 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604350 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604304 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604386 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604438 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.604834 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.604810 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4617ca5d-ea03-492c-a926-924ddf3b5860-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.606551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.606525 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.607132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.607113 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.665468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.665392 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4617ca5d-ea03-492c-a926-924ddf3b5860-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.665646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.665474 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mk28\" (UniqueName: \"kubernetes.io/projected/4617ca5d-ea03-492c-a926-924ddf3b5860-kube-api-access-6mk28\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd\" (UID: \"4617ca5d-ea03-492c-a926-924ddf3b5860\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:14.690330 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:14.690298 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:15.183540 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.183377 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd"] Nov 01 00:34:15.186529 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:34:15.186501 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4617ca5d_ea03_492c_a926_924ddf3b5860.slice/crio-b6dac111214be981f15b03d0eb6a8471be4388f7bbc8397d92adb69c8502add5 WatchSource:0}: Error finding container b6dac111214be981f15b03d0eb6a8471be4388f7bbc8397d92adb69c8502add5: Status 404 returned error can't find the container with id b6dac111214be981f15b03d0eb6a8471be4388f7bbc8397d92adb69c8502add5 Nov 01 00:34:15.189321 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.189295 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:34:15.189409 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.189353 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:34:15.189409 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.189381 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:34:15.457717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.457596 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" event={"ID":"4cda9f88-23b8-4d10-98b9-de79a789d559","Type":"ContainerStarted","Data":"f5cf6890ed41adf5a729a59ab60a1321dfdcc49dca09275b761d1f6f137f8832"} Nov 01 00:34:15.458213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.457826 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" Nov 01 00:34:15.459474 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.459448 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" event={"ID":"4617ca5d-ea03-492c-a926-924ddf3b5860","Type":"ContainerStarted","Data":"272234517bf899cea46c41cf7f78e7dbe432d8200daf7b486ee85c43685b4fff"} Nov 01 00:34:15.459601 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.459487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" event={"ID":"4617ca5d-ea03-492c-a926-924ddf3b5860","Type":"ContainerStarted","Data":"b6dac111214be981f15b03d0eb6a8471be4388f7bbc8397d92adb69c8502add5"} Nov 01 00:34:15.576092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.576036 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" podStartSLOduration=1.7417903240000001 podStartE2EDuration="7.576020841s" totalImagesPullingTime="5.834230517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:15.574228205 +0000 UTC m=+645.553810540" watchObservedRunningTime="2026-11-01 00:34:15.576020841 +0000 UTC m=+645.555603175" Nov 01 00:34:15.691107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.690991 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:15.698916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:15.698864 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" podStartSLOduration=1.6988520660000002 podStartE2EDuration="1.698852066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:15.697462236 +0000 UTC m=+645.677044591" watchObservedRunningTime="2026-11-01 00:34:15.698852066 +0000 UTC m=+645.678434402" Nov 01 00:34:16.695885 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:16.695854 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:17.472276 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:17.472245 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:17.473199 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:17.473176 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd" Nov 01 00:34:21.493982 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:21.493949 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" event={"ID":"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744","Type":"ContainerStarted","Data":"0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14"} Nov 01 00:34:21.494396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:21.494049 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:21.712941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:21.712297 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" podStartSLOduration=1.396116674 podStartE2EDuration="15.712279143s" totalImagesPullingTime="14.316162469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:21.71221424 +0000 UTC m=+651.691796582" watchObservedRunningTime="2026-11-01 00:34:21.712279143 +0000 UTC m=+651.691861479" Nov 01 00:34:26.467774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:26.467746 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tx8vx" Nov 01 00:34:32.501675 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:32.501642 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 00:34:34.887130 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:34.887096 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mmn2t"] Nov 01 00:34:34.895345 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:34.895320 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:34.948399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:34.948372 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mmn2t"] Nov 01 00:34:34.984203 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:34.984178 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflf9\" (UniqueName: \"kubernetes.io/projected/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-kube-api-access-zflf9\") pod \"limitador-limitador-74dfcfb894-mmn2t\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:34.984330 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:34.984305 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-config-file\") pod \"limitador-limitador-74dfcfb894-mmn2t\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:34.995813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:34.995786 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gp6r7\"" Nov 01 00:34:34.995813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:34.995795 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 00:34:35.085607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:35.085581 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-config-file\") pod \"limitador-limitador-74dfcfb894-mmn2t\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:35.085774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:35.085645 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflf9\" (UniqueName: \"kubernetes.io/projected/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-kube-api-access-zflf9\") pod \"limitador-limitador-74dfcfb894-mmn2t\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:35.086179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:35.086161 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-config-file\") pod \"limitador-limitador-74dfcfb894-mmn2t\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:35.137993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:35.137923 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mmn2t"] Nov 01 00:34:35.146439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:35.146418 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflf9\" (UniqueName: \"kubernetes.io/projected/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-kube-api-access-zflf9\") pod \"limitador-limitador-74dfcfb894-mmn2t\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:35.208494 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:35.208466 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:35.585718 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:35.585282 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mmn2t"] Nov 01 00:34:35.587323 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:34:35.587279 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4d1c1f_c4d3_4c8b_bf62_51450aeed7ff.slice/crio-405de3d6e30447b08fb0dc651a2aae222f149cc9281fc815baa2e503f7e928eb WatchSource:0}: Error finding container 405de3d6e30447b08fb0dc651a2aae222f149cc9281fc815baa2e503f7e928eb: Status 404 returned error can't find the container with id 405de3d6e30447b08fb0dc651a2aae222f149cc9281fc815baa2e503f7e928eb Nov 01 00:34:36.573469 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:36.573434 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" event={"ID":"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff","Type":"ContainerStarted","Data":"405de3d6e30447b08fb0dc651a2aae222f149cc9281fc815baa2e503f7e928eb"} Nov 01 00:34:38.584672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:38.584616 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" event={"ID":"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff","Type":"ContainerStarted","Data":"cbf7ff57fee80b3d11eff0cf0b9e939f67fd66eb58ac6c274db98308b8038130"} Nov 01 00:34:38.585064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:38.584778 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:38.701671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:38.701604 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" podStartSLOduration=2.191518598 podStartE2EDuration="4.70158895s" totalImagesPullingTime="2.510070352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:34:38.699794382 +0000 UTC m=+668.679376721" watchObservedRunningTime="2026-11-01 00:34:38.70158895 +0000 UTC m=+668.681171286" Nov 01 00:34:43.102771 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:34:43.102738 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:34:49.590789 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:49.590763 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:52.919350 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:52.919318 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mmn2t"] Nov 01 00:34:52.919751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:52.919534 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" podUID="fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" containerName="limitador" containerID="cri-o://cbf7ff57fee80b3d11eff0cf0b9e939f67fd66eb58ac6c274db98308b8038130" gracePeriod=30 Nov 01 00:34:53.664614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:53.664582 4910 generic.go:356] "Generic (PLEG): container finished" podID="fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" containerID="cbf7ff57fee80b3d11eff0cf0b9e939f67fd66eb58ac6c274db98308b8038130" exitCode=0 Nov 01 00:34:53.664786 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:53.664642 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" event={"ID":"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff","Type":"ContainerDied","Data":"cbf7ff57fee80b3d11eff0cf0b9e939f67fd66eb58ac6c274db98308b8038130"} Nov 01 00:34:53.887414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:53.887389 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:53.926244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:53.926181 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-config-file\" (UniqueName: \"kubernetes.io/configmap/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-config-file\") pod \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " Nov 01 00:34:53.926244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:53.926229 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-kube-api-access-zflf9\" (UniqueName: \"kubernetes.io/projected/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-kube-api-access-zflf9\") pod \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\" (UID: \"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff\") " Nov 01 00:34:53.926733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:53.926572 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-config-file" pod "fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" (UID: "fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:34:53.928459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:53.928438 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-kube-api-access-zflf9" pod "fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" (UID: "fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff"). InnerVolumeSpecName "kube-api-access-zflf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:34:54.026797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:54.026769 4910 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-config-file\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:34:54.026797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:54.026792 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zflf9\" (UniqueName: \"kubernetes.io/projected/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff-kube-api-access-zflf9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:34:54.671353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:54.671318 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" event={"ID":"fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff","Type":"ContainerDied","Data":"405de3d6e30447b08fb0dc651a2aae222f149cc9281fc815baa2e503f7e928eb"} Nov 01 00:34:54.671508 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:54.671400 4910 scope.go:122] "RemoveContainer" containerID="cbf7ff57fee80b3d11eff0cf0b9e939f67fd66eb58ac6c274db98308b8038130" Nov 01 00:34:54.671508 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:54.671336 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mmn2t" Nov 01 00:34:54.835351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:54.835322 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mmn2t"] Nov 01 00:34:54.886340 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:54.886305 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mmn2t"] Nov 01 00:34:56.190335 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:34:56.190304 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" path="/var/lib/kubelet/pods/fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff/volumes" Nov 01 00:35:00.815262 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.815234 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq"] Nov 01 00:35:00.815744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.815651 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" containerName="limitador" Nov 01 00:35:00.815744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.815663 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" containerName="limitador" Nov 01 00:35:00.815744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.815745 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4d1c1f-c4d3-4c8b-bf62-51450aeed7ff" containerName="limitador" Nov 01 00:35:00.818177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.818160 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:00.873755 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.873719 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 00:35:00.873755 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.873736 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 00:35:00.876163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.876134 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq"] Nov 01 00:35:00.882183 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.882160 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqqb\" (UniqueName: \"kubernetes.io/projected/7ad168f5-aacb-4793-87a6-088bd89161ec-kube-api-access-mqqqb\") pod \"postgres-7cdbf6dc44-6lcwq\" (UID: \"7ad168f5-aacb-4793-87a6-088bd89161ec\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:00.882298 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.882279 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7ad168f5-aacb-4793-87a6-088bd89161ec-data\") pod \"postgres-7cdbf6dc44-6lcwq\" (UID: \"7ad168f5-aacb-4793-87a6-088bd89161ec\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:00.923616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.923590 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-b64l9\"" Nov 01 00:35:00.923616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.923597 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 00:35:00.983312 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.983283 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqqb\" (UniqueName: \"kubernetes.io/projected/7ad168f5-aacb-4793-87a6-088bd89161ec-kube-api-access-mqqqb\") pod \"postgres-7cdbf6dc44-6lcwq\" (UID: \"7ad168f5-aacb-4793-87a6-088bd89161ec\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:00.983470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.983339 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7ad168f5-aacb-4793-87a6-088bd89161ec-data\") pod \"postgres-7cdbf6dc44-6lcwq\" (UID: \"7ad168f5-aacb-4793-87a6-088bd89161ec\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:00.983761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:00.983743 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7ad168f5-aacb-4793-87a6-088bd89161ec-data\") pod \"postgres-7cdbf6dc44-6lcwq\" (UID: \"7ad168f5-aacb-4793-87a6-088bd89161ec\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:01.044509 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:01.044482 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqqb\" (UniqueName: \"kubernetes.io/projected/7ad168f5-aacb-4793-87a6-088bd89161ec-kube-api-access-mqqqb\") pod \"postgres-7cdbf6dc44-6lcwq\" (UID: \"7ad168f5-aacb-4793-87a6-088bd89161ec\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:01.130031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:01.129962 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:01.498707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:01.498684 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq"] Nov 01 00:35:01.500537 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:35:01.500508 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad168f5_aacb_4793_87a6_088bd89161ec.slice/crio-bf303e765a3346bb6f7b1b6b4a8eabd08ad55626c4c9a4656309aa64ba81aab8 WatchSource:0}: Error finding container bf303e765a3346bb6f7b1b6b4a8eabd08ad55626c4c9a4656309aa64ba81aab8: Status 404 returned error can't find the container with id bf303e765a3346bb6f7b1b6b4a8eabd08ad55626c4c9a4656309aa64ba81aab8 Nov 01 00:35:01.709236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:01.709200 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" event={"ID":"7ad168f5-aacb-4793-87a6-088bd89161ec","Type":"ContainerStarted","Data":"bf303e765a3346bb6f7b1b6b4a8eabd08ad55626c4c9a4656309aa64ba81aab8"} Nov 01 00:35:08.751309 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:08.751272 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" event={"ID":"7ad168f5-aacb-4793-87a6-088bd89161ec","Type":"ContainerStarted","Data":"b4f385d96ef29d48127fc1d7c413faee0b758d93c24eafe937c1121a38f8fce9"} Nov 01 00:35:08.751742 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:08.751410 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:08.866494 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:08.866438 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" podStartSLOduration=1.897803943 podStartE2EDuration="8.866418433s" totalImagesPullingTime="6.96861449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:08.86609476 +0000 UTC m=+698.845677096" watchObservedRunningTime="2026-11-01 00:35:08.866418433 +0000 UTC m=+698.846000774" Nov 01 00:35:13.101014 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:35:13.100979 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:35:14.786390 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:14.786361 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6lcwq" Nov 01 00:35:21.717389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.717348 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-tmtkx"] Nov 01 00:35:21.720968 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.720947 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.773808 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.773789 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 00:35:21.773921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.773820 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 00:35:21.823694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.823663 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 00:35:21.823694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.823688 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 00:35:21.823903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.823868 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wgq79\"" Nov 01 00:35:21.823961 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.823916 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 00:35:21.826354 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.826334 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-tmtkx"] Nov 01 00:35:21.862980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.862956 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-webhook-cert\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.863082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.862994 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-metrics-certs\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.863082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.863044 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q88x\" (UniqueName: \"kubernetes.io/projected/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-kube-api-access-4q88x\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.963952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.963916 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-webhook-cert\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.963952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.963952 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-metrics-certs\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.964173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.964085 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q88x\" (UniqueName: \"kubernetes.io/projected/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-kube-api-access-4q88x\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.966593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.966572 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-metrics-certs\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:21.966808 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:21.966786 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-webhook-cert\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:22.027822 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.027792 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q88x\" (UniqueName: \"kubernetes.io/projected/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-kube-api-access-4q88x\") pod \"maas-controller-579cf7c6d-tmtkx\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:22.033673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.033634 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:22.119676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.119644 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-q25wr"] Nov 01 00:35:22.123429 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.123409 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.183958 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.183934 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-q25wr"] Nov 01 00:35:22.265727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.265700 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.265914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.265746 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.265914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.265842 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktq2\" (UniqueName: \"kubernetes.io/projected/b4a8bae4-5f29-4776-a979-eb6d0532e820-kube-api-access-pktq2\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.366849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.366773 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.366849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.366834 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.367029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.366889 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktq2\" (UniqueName: \"kubernetes.io/projected/b4a8bae4-5f29-4776-a979-eb6d0532e820-kube-api-access-pktq2\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.369332 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.369307 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-metrics-certs\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.369439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.369331 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-webhook-cert\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.407009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.406988 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-tmtkx"] Nov 01 00:35:22.408328 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:35:22.408302 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205ad4ee_bfcb_4c12_a4db_68fe9ce80d4d.slice/crio-8ea266a665bf51f9d62882ee9b28697bcbd0ccd8939e16c58c30beeeb3f0ff4e WatchSource:0}: Error finding container 8ea266a665bf51f9d62882ee9b28697bcbd0ccd8939e16c58c30beeeb3f0ff4e: Status 404 returned error can't find the container with id 8ea266a665bf51f9d62882ee9b28697bcbd0ccd8939e16c58c30beeeb3f0ff4e Nov 01 00:35:22.427985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.427966 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktq2\" (UniqueName: \"kubernetes.io/projected/b4a8bae4-5f29-4776-a979-eb6d0532e820-kube-api-access-pktq2\") pod \"maas-controller-5fb6cd4f9b-q25wr\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.436825 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.436803 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:22.808447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.808421 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-q25wr"] Nov 01 00:35:22.810181 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:35:22.810156 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a8bae4_5f29_4776_a979_eb6d0532e820.slice/crio-87eba920101707c482c09b4bdd5f3141642901785ebe9152fe14cbac326f4f58 WatchSource:0}: Error finding container 87eba920101707c482c09b4bdd5f3141642901785ebe9152fe14cbac326f4f58: Status 404 returned error can't find the container with id 87eba920101707c482c09b4bdd5f3141642901785ebe9152fe14cbac326f4f58 Nov 01 00:35:22.829333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.829297 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" event={"ID":"b4a8bae4-5f29-4776-a979-eb6d0532e820","Type":"ContainerStarted","Data":"87eba920101707c482c09b4bdd5f3141642901785ebe9152fe14cbac326f4f58"} Nov 01 00:35:22.830459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:22.830433 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" event={"ID":"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d","Type":"ContainerStarted","Data":"8ea266a665bf51f9d62882ee9b28697bcbd0ccd8939e16c58c30beeeb3f0ff4e"} Nov 01 00:35:27.860163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:27.860129 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" event={"ID":"b4a8bae4-5f29-4776-a979-eb6d0532e820","Type":"ContainerStarted","Data":"7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637"} Nov 01 00:35:27.860493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:27.860209 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:28.019985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:28.019929 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" podStartSLOduration=1.039189898 podStartE2EDuration="6.01991068s" totalImagesPullingTime="4.980720782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:28.018594712 +0000 UTC m=+717.998177047" watchObservedRunningTime="2026-11-01 00:35:28.01991068 +0000 UTC m=+717.999493016" Nov 01 00:35:28.866754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:28.866717 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" event={"ID":"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d","Type":"ContainerStarted","Data":"e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9"} Nov 01 00:35:28.867184 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:28.866878 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:28.986704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:28.986654 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" podStartSLOduration=2.56205551 podStartE2EDuration="7.986639589s" totalImagesPullingTime="5.424584079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:28.984506542 +0000 UTC m=+718.964089056" watchObservedRunningTime="2026-11-01 00:35:28.986639589 +0000 UTC m=+718.966221920" Nov 01 00:35:31.172639 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.172600 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf"] Nov 01 00:35:31.177550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.177527 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.231912 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.231873 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 00:35:31.234744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.234721 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf"] Nov 01 00:35:31.282043 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.282010 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 00:35:31.282189 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.282057 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 00:35:31.282189 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.282127 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-v2g2n\"" Nov 01 00:35:31.348472 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.348442 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hwl\" (UniqueName: \"kubernetes.io/projected/e6308d92-d7fd-4085-a4dc-7a1971245c25-kube-api-access-f8hwl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.348606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.348490 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6308d92-d7fd-4085-a4dc-7a1971245c25-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.348606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.348596 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.348707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.348637 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.449672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.449579 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.449672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.449656 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hwl\" (UniqueName: \"kubernetes.io/projected/e6308d92-d7fd-4085-a4dc-7a1971245c25-kube-api-access-f8hwl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.449871 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.449688 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6308d92-d7fd-4085-a4dc-7a1971245c25-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.449871 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:35:31.449696 4910 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 01 00:35:31.449871 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.449723 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.449871 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:35:31.449780 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-proxy-source podName:e6308d92-d7fd-4085-a4dc-7a1971245c25 nodeName:}" failed. No retries permitted until 2026-11-01 00:35:31.949758417 +0000 UTC m=+721.929340731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" (UID: "e6308d92-d7fd-4085-a4dc-7a1971245c25") : configmap "usage-tenancy-proxy-source" not found Nov 01 00:35:31.450302 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.450275 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.452170 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.452154 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6308d92-d7fd-4085-a4dc-7a1971245c25-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.511271 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.511244 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hwl\" (UniqueName: \"kubernetes.io/projected/e6308d92-d7fd-4085-a4dc-7a1971245c25-kube-api-access-f8hwl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.953658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.953607 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:31.954359 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:31.954340 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e6308d92-d7fd-4085-a4dc-7a1971245c25-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-w2ljf\" (UID: \"e6308d92-d7fd-4085-a4dc-7a1971245c25\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:32.089838 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:32.089812 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:32.463975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:32.463946 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf"] Nov 01 00:35:32.465702 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:35:32.465673 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6308d92_d7fd_4085_a4dc_7a1971245c25.slice/crio-1b8f94cf7105d749fbdd9b2135e277f428dfe6c75e1089f5d7fbf5f36cca992e WatchSource:0}: Error finding container 1b8f94cf7105d749fbdd9b2135e277f428dfe6c75e1089f5d7fbf5f36cca992e: Status 404 returned error can't find the container with id 1b8f94cf7105d749fbdd9b2135e277f428dfe6c75e1089f5d7fbf5f36cca992e Nov 01 00:35:32.893574 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:32.893532 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" event={"ID":"e6308d92-d7fd-4085-a4dc-7a1971245c25","Type":"ContainerStarted","Data":"1b8f94cf7105d749fbdd9b2135e277f428dfe6c75e1089f5d7fbf5f36cca992e"} Nov 01 00:35:38.869146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:38.869113 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:35:39.051874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.051836 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-tmtkx"] Nov 01 00:35:39.052121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.052090 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" podUID="205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" containerName="manager" containerID="cri-o://e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9" gracePeriod=30 Nov 01 00:35:39.054223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.054194 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:39.343969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.343945 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:39.528211 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.528174 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-kube-api-access-4q88x\" (UniqueName: \"kubernetes.io/projected/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-kube-api-access-4q88x\") pod \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " Nov 01 00:35:39.528406 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.528246 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-metrics-certs\") pod \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " Nov 01 00:35:39.528692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.528665 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-webhook-cert\") pod \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\" (UID: \"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d\") " Nov 01 00:35:39.530784 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.530757 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-metrics-certs" pod "205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" (UID: "205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:35:39.531464 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.531437 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-kube-api-access-4q88x" pod "205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" (UID: "205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d"). InnerVolumeSpecName "kube-api-access-4q88x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:35:39.534909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.534884 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-webhook-cert" pod "205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" (UID: "205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:35:39.630465 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.630431 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q88x\" (UniqueName: \"kubernetes.io/projected/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-kube-api-access-4q88x\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:35:39.630465 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.630470 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-metrics-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:35:39.630728 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.630485 4910 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d-webhook-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:35:39.948037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.947895 4910 generic.go:356] "Generic (PLEG): container finished" podID="205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" containerID="e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9" exitCode=0 Nov 01 00:35:39.948037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.947989 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" Nov 01 00:35:39.948550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.948035 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" event={"ID":"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d","Type":"ContainerDied","Data":"e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9"} Nov 01 00:35:39.948550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.948117 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-tmtkx" event={"ID":"205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d","Type":"ContainerDied","Data":"8ea266a665bf51f9d62882ee9b28697bcbd0ccd8939e16c58c30beeeb3f0ff4e"} Nov 01 00:35:39.948550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.948150 4910 scope.go:122] "RemoveContainer" containerID="e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9" Nov 01 00:35:39.961059 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.961037 4910 scope.go:122] "RemoveContainer" containerID="e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9" Nov 01 00:35:39.961393 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:35:39.961368 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9\": container with ID starting with e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9 not found: ID does not exist" containerID="e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9" Nov 01 00:35:39.961483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:39.961408 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9"} err="failed to get container status \"e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9\": rpc error: code = NotFound desc = could not find container \"e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9\": container with ID starting with e65b91a97f1db85ccf06a1383eaf0db9204b487b95a468bc32d7677d48210bf9 not found: ID does not exist" Nov 01 00:35:40.114531 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:40.114467 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-tmtkx"] Nov 01 00:35:40.164068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:40.164031 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-tmtkx"] Nov 01 00:35:40.191639 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:40.191599 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" path="/var/lib/kubelet/pods/205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d/volumes" Nov 01 00:35:43.108561 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:35:43.108525 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:35:45.986562 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:45.986524 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" event={"ID":"e6308d92-d7fd-4085-a4dc-7a1971245c25","Type":"ContainerStarted","Data":"0516e1de8985b86106ab53cd799c85cc0040abbee925c1b85d135d01e49dc8e3"} Nov 01 00:35:45.986955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:45.986656 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:46.105356 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:46.105310 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" podStartSLOduration=2.275410604 podStartE2EDuration="15.105295817s" totalImagesPullingTime="12.829885213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:46.103892865 +0000 UTC m=+736.083475201" watchObservedRunningTime="2026-11-01 00:35:46.105295817 +0000 UTC m=+736.084878179" Nov 01 00:35:49.447027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.446993 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sdwtw"] Nov 01 00:35:49.447439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.447367 4910 cpu_manager.go:414] "containerMap: removing container" podUID="205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" containerName="manager" Nov 01 00:35:49.447439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.447377 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" containerName="manager" Nov 01 00:35:49.447522 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.447445 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="205ad4ee-bfcb-4c12-a4db-68fe9ce80d4d" containerName="manager" Nov 01 00:35:49.450673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.450658 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.516810 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.516770 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c2335f-428b-4a64-9f23-bac748875f57-cert\") pod \"kserve-controller-manager-776cc66d47-sdwtw\" (UID: \"a0c2335f-428b-4a64-9f23-bac748875f57\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.516991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.516915 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsq2\" (UniqueName: \"kubernetes.io/projected/a0c2335f-428b-4a64-9f23-bac748875f57-kube-api-access-ntsq2\") pod \"kserve-controller-manager-776cc66d47-sdwtw\" (UID: \"a0c2335f-428b-4a64-9f23-bac748875f57\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.555501 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.555468 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-snvj8\"" Nov 01 00:35:49.555501 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.555469 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 00:35:49.557924 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.557896 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sdwtw"] Nov 01 00:35:49.560220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.560199 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd"] Nov 01 00:35:49.563785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.563770 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:49.616380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.616358 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 00:35:49.617677 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.617648 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c2335f-428b-4a64-9f23-bac748875f57-cert\") pod \"kserve-controller-manager-776cc66d47-sdwtw\" (UID: \"a0c2335f-428b-4a64-9f23-bac748875f57\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.617784 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.617686 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24z5v\" (UniqueName: \"kubernetes.io/projected/8fb01bf6-8775-4643-92df-fd1744163a64-kube-api-access-24z5v\") pod \"llmisvc-controller-manager-869d5dcc5d-978dd\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:49.617784 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.617775 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsq2\" (UniqueName: \"kubernetes.io/projected/a0c2335f-428b-4a64-9f23-bac748875f57-kube-api-access-ntsq2\") pod \"kserve-controller-manager-776cc66d47-sdwtw\" (UID: \"a0c2335f-428b-4a64-9f23-bac748875f57\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.618099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.617885 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fb01bf6-8775-4643-92df-fd1744163a64-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-978dd\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:49.618858 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.618836 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kzff5\"" Nov 01 00:35:49.621140 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.621116 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c2335f-428b-4a64-9f23-bac748875f57-cert\") pod \"kserve-controller-manager-776cc66d47-sdwtw\" (UID: \"a0c2335f-428b-4a64-9f23-bac748875f57\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.622128 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.622111 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd"] Nov 01 00:35:49.680556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.680523 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsq2\" (UniqueName: \"kubernetes.io/projected/a0c2335f-428b-4a64-9f23-bac748875f57-kube-api-access-ntsq2\") pod \"kserve-controller-manager-776cc66d47-sdwtw\" (UID: \"a0c2335f-428b-4a64-9f23-bac748875f57\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.718666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.718584 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fb01bf6-8775-4643-92df-fd1744163a64-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-978dd\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:49.718666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.718636 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24z5v\" (UniqueName: \"kubernetes.io/projected/8fb01bf6-8775-4643-92df-fd1744163a64-kube-api-access-24z5v\") pod \"llmisvc-controller-manager-869d5dcc5d-978dd\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:49.721210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.721185 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fb01bf6-8775-4643-92df-fd1744163a64-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-978dd\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:49.763577 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.763546 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:49.779914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.779889 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24z5v\" (UniqueName: \"kubernetes.io/projected/8fb01bf6-8775-4643-92df-fd1744163a64-kube-api-access-24z5v\") pod \"llmisvc-controller-manager-869d5dcc5d-978dd\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:49.876765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:49.876737 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:50.146794 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:50.146769 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sdwtw"] Nov 01 00:35:50.148790 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:35:50.148755 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c2335f_428b_4a64_9f23_bac748875f57.slice/crio-b03bcaf1ac118b374ed5220fc5634fc07ef95de6d8143055e81d36773d960a51 WatchSource:0}: Error finding container b03bcaf1ac118b374ed5220fc5634fc07ef95de6d8143055e81d36773d960a51: Status 404 returned error can't find the container with id b03bcaf1ac118b374ed5220fc5634fc07ef95de6d8143055e81d36773d960a51 Nov 01 00:35:50.250123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:50.250095 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd"] Nov 01 00:35:50.251277 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:35:50.251250 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8fb01bf6_8775_4643_92df_fd1744163a64.slice/crio-7e7f95ddc72b743b781da2e47c1586b66ed7678c23171fada4537139e85003fd WatchSource:0}: Error finding container 7e7f95ddc72b743b781da2e47c1586b66ed7678c23171fada4537139e85003fd: Status 404 returned error can't find the container with id 7e7f95ddc72b743b781da2e47c1586b66ed7678c23171fada4537139e85003fd Nov 01 00:35:51.014441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:51.014401 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" event={"ID":"a0c2335f-428b-4a64-9f23-bac748875f57","Type":"ContainerStarted","Data":"b03bcaf1ac118b374ed5220fc5634fc07ef95de6d8143055e81d36773d960a51"} Nov 01 00:35:51.015774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:51.015743 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" event={"ID":"8fb01bf6-8775-4643-92df-fd1744163a64","Type":"ContainerStarted","Data":"7e7f95ddc72b743b781da2e47c1586b66ed7678c23171fada4537139e85003fd"} Nov 01 00:35:54.037712 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:54.037681 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" event={"ID":"8fb01bf6-8775-4643-92df-fd1744163a64","Type":"ContainerStarted","Data":"e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1"} Nov 01 00:35:54.038090 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:54.037829 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:35:54.154590 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:54.154539 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" podStartSLOduration=1.861396289 podStartE2EDuration="5.154524775s" totalImagesPullingTime="3.293128486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:54.153466741 +0000 UTC m=+744.133049071" watchObservedRunningTime="2026-11-01 00:35:54.154524775 +0000 UTC m=+744.134107120" Nov 01 00:35:56.997825 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:56.997795 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-w2ljf" Nov 01 00:35:57.060507 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:57.060473 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" event={"ID":"a0c2335f-428b-4a64-9f23-bac748875f57","Type":"ContainerStarted","Data":"476aa5e5a12520d69c5b698cbb08f2652eda24340a18d2f3739454bd1df4a61c"} Nov 01 00:35:57.060694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:57.060600 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:35:57.231020 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:35:57.230968 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" podStartSLOduration=2.203140276 podStartE2EDuration="8.230954368s" totalImagesPullingTime="6.027814092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:35:57.229518601 +0000 UTC m=+747.209100936" watchObservedRunningTime="2026-11-01 00:35:57.230954368 +0000 UTC m=+747.210536702" Nov 01 00:36:13.102862 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:36:13.102826 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:36:19.138570 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.130278 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hg4hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": read tcp 10.132.0.2:52136->10.132.0.36:15021: read: connection reset by peer" start-of-body= Nov 01 00:36:19.138570 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.130340 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" podUID="c16617ec-fda2-4f67-853b-fd58ff568d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": read tcp 10.132.0.2:52136->10.132.0.36:15021: read: connection reset by peer" Nov 01 00:36:19.205502 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.205468 4910 generic.go:356] "Generic (PLEG): container finished" podID="c16617ec-fda2-4f67-853b-fd58ff568d2c" containerID="607f39218bae366afdb1e3fc9eb6466c33c6d59243d2a523d394be75b7311efe" exitCode=0 Nov 01 00:36:19.205697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.205579 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" event={"ID":"c16617ec-fda2-4f67-853b-fd58ff568d2c","Type":"ContainerDied","Data":"607f39218bae366afdb1e3fc9eb6466c33c6d59243d2a523d394be75b7311efe"} Nov 01 00:36:19.206005 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.205988 4910 scope.go:122] "RemoveContainer" containerID="607f39218bae366afdb1e3fc9eb6466c33c6d59243d2a523d394be75b7311efe" Nov 01 00:36:19.207080 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.207050 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:36:19.207134 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.207119 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:36:19.207172 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:19.207153 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:36:20.213037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:20.213004 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" event={"ID":"c16617ec-fda2-4f67-853b-fd58ff568d2c","Type":"ContainerStarted","Data":"af8d3266b9bf16c62a595a0a8bf3a71ee06000e53e858de13cb763485ea8c12c"} Nov 01 00:36:21.170323 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:21.170233 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:36:21.170616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:21.170594 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hg4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:36:21.170817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:21.170791 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" podUID="c16617ec-fda2-4f67-853b-fd58ff568d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 01 00:36:22.170536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:22.170066 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hg4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:36:22.170536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:22.170129 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" podUID="c16617ec-fda2-4f67-853b-fd58ff568d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 01 00:36:23.170214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:23.170171 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hg4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:36:23.170494 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:23.170467 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" podUID="c16617ec-fda2-4f67-853b-fd58ff568d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 01 00:36:24.174675 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:24.174053 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hg4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:36:24.174675 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:24.174114 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" podUID="c16617ec-fda2-4f67-853b-fd58ff568d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 01 00:36:25.046059 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:25.046026 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:36:25.170161 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:25.170129 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hg4hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 01 00:36:25.170356 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:25.170203 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" podUID="c16617ec-fda2-4f67-853b-fd58ff568d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 01 00:36:26.172870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:26.172839 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:36:26.173268 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:26.173226 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:36:26.173937 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:26.173918 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hg4hr" Nov 01 00:36:28.071081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:36:28.071054 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-sdwtw" Nov 01 00:36:43.101985 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:36:43.101949 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:37:00.572914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.572870 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6"] Nov 01 00:37:00.576850 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.576827 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:00.634674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.634618 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6"] Nov 01 00:37:00.648209 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.648176 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8469e4e1-96b0-4fce-a880-eaa35ff0157a-cert\") pod \"llmisvc-controller-manager-6466c554d8-ng9v6\" (UID: \"8469e4e1-96b0-4fce-a880-eaa35ff0157a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:00.648378 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.648320 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wwm\" (UniqueName: \"kubernetes.io/projected/8469e4e1-96b0-4fce-a880-eaa35ff0157a-kube-api-access-f6wwm\") pod \"llmisvc-controller-manager-6466c554d8-ng9v6\" (UID: \"8469e4e1-96b0-4fce-a880-eaa35ff0157a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:00.749635 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.749584 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8469e4e1-96b0-4fce-a880-eaa35ff0157a-cert\") pod \"llmisvc-controller-manager-6466c554d8-ng9v6\" (UID: \"8469e4e1-96b0-4fce-a880-eaa35ff0157a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:00.749819 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.749673 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wwm\" (UniqueName: \"kubernetes.io/projected/8469e4e1-96b0-4fce-a880-eaa35ff0157a-kube-api-access-f6wwm\") pod \"llmisvc-controller-manager-6466c554d8-ng9v6\" (UID: \"8469e4e1-96b0-4fce-a880-eaa35ff0157a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:00.752160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.752132 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8469e4e1-96b0-4fce-a880-eaa35ff0157a-cert\") pod \"llmisvc-controller-manager-6466c554d8-ng9v6\" (UID: \"8469e4e1-96b0-4fce-a880-eaa35ff0157a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:00.811159 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.811127 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wwm\" (UniqueName: \"kubernetes.io/projected/8469e4e1-96b0-4fce-a880-eaa35ff0157a-kube-api-access-f6wwm\") pod \"llmisvc-controller-manager-6466c554d8-ng9v6\" (UID: \"8469e4e1-96b0-4fce-a880-eaa35ff0157a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:00.890033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:00.889956 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:01.265485 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:37:01.265451 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8469e4e1_96b0_4fce_a880_eaa35ff0157a.slice/crio-8bb8e8cca52fbed32c5be89632edf7b467ca864edb0217c4e89c14b971c19f5c WatchSource:0}: Error finding container 8bb8e8cca52fbed32c5be89632edf7b467ca864edb0217c4e89c14b971c19f5c: Status 404 returned error can't find the container with id 8bb8e8cca52fbed32c5be89632edf7b467ca864edb0217c4e89c14b971c19f5c Nov 01 00:37:01.270330 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:01.270303 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6"] Nov 01 00:37:01.442419 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:01.442385 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" event={"ID":"8469e4e1-96b0-4fce-a880-eaa35ff0157a","Type":"ContainerStarted","Data":"8bb8e8cca52fbed32c5be89632edf7b467ca864edb0217c4e89c14b971c19f5c"} Nov 01 00:37:02.450312 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:02.450223 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" event={"ID":"8469e4e1-96b0-4fce-a880-eaa35ff0157a","Type":"ContainerStarted","Data":"5d04343fafba5cea476d18c1b7fd2d9e1a22ebf7a07643ba4ec0a87e613e9404"} Nov 01 00:37:02.450312 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:02.450291 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:02.568533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:02.568474 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" podStartSLOduration=1.784644192 podStartE2EDuration="2.568460444s" totalImagesPullingTime="783.816252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:37:02.567906151 +0000 UTC m=+812.547488488" watchObservedRunningTime="2026-11-01 00:37:02.568460444 +0000 UTC m=+812.548042779" Nov 01 00:37:13.102794 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:37:13.102760 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:37:33.458181 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:33.458151 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ng9v6" Nov 01 00:37:33.628238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:33.628204 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd"] Nov 01 00:37:33.628520 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:33.628461 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" podUID="8fb01bf6-8775-4643-92df-fd1744163a64" containerName="manager" containerID="cri-o://e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1" gracePeriod=30 Nov 01 00:37:33.879834 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:33.879811 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:37:34.033183 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.033059 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb01bf6-8775-4643-92df-fd1744163a64-kube-api-access-24z5v\" (UniqueName: \"kubernetes.io/projected/8fb01bf6-8775-4643-92df-fd1744163a64-kube-api-access-24z5v\") pod \"8fb01bf6-8775-4643-92df-fd1744163a64\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " Nov 01 00:37:34.033326 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.033219 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb01bf6-8775-4643-92df-fd1744163a64-cert\" (UniqueName: \"kubernetes.io/secret/8fb01bf6-8775-4643-92df-fd1744163a64-cert\") pod \"8fb01bf6-8775-4643-92df-fd1744163a64\" (UID: \"8fb01bf6-8775-4643-92df-fd1744163a64\") " Nov 01 00:37:34.035291 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.035261 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb01bf6-8775-4643-92df-fd1744163a64-kube-api-access-24z5v" pod "8fb01bf6-8775-4643-92df-fd1744163a64" (UID: "8fb01bf6-8775-4643-92df-fd1744163a64"). InnerVolumeSpecName "kube-api-access-24z5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:37:34.035404 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.035329 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb01bf6-8775-4643-92df-fd1744163a64-cert" pod "8fb01bf6-8775-4643-92df-fd1744163a64" (UID: "8fb01bf6-8775-4643-92df-fd1744163a64"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:37:34.133938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.133906 4910 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fb01bf6-8775-4643-92df-fd1744163a64-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:37:34.133938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.133933 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24z5v\" (UniqueName: \"kubernetes.io/projected/8fb01bf6-8775-4643-92df-fd1744163a64-kube-api-access-24z5v\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:37:34.626392 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.626357 4910 generic.go:356] "Generic (PLEG): container finished" podID="8fb01bf6-8775-4643-92df-fd1744163a64" containerID="e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1" exitCode=0 Nov 01 00:37:34.626920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.626409 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" Nov 01 00:37:34.626920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.626429 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" event={"ID":"8fb01bf6-8775-4643-92df-fd1744163a64","Type":"ContainerDied","Data":"e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1"} Nov 01 00:37:34.626920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.626471 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd" event={"ID":"8fb01bf6-8775-4643-92df-fd1744163a64","Type":"ContainerDied","Data":"7e7f95ddc72b743b781da2e47c1586b66ed7678c23171fada4537139e85003fd"} Nov 01 00:37:34.626920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.626491 4910 scope.go:122] "RemoveContainer" containerID="e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1" Nov 01 00:37:34.635791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.635617 4910 scope.go:122] "RemoveContainer" containerID="e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1" Nov 01 00:37:34.635913 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:37:34.635896 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1\": container with ID starting with e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1 not found: ID does not exist" containerID="e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1" Nov 01 00:37:34.635964 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.635922 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1"} err="failed to get container status \"e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1\": rpc error: code = NotFound desc = could not find container \"e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1\": container with ID starting with e194881f96f3259dfe4f5a47caed6b7b92af4ed70a9dbf7052cd37a3d87954d1 not found: ID does not exist" Nov 01 00:37:34.791520 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.791487 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd"] Nov 01 00:37:34.841363 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:34.841334 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-978dd"] Nov 01 00:37:36.189852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:36.189822 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb01bf6-8775-4643-92df-fd1744163a64" path="/var/lib/kubelet/pods/8fb01bf6-8775-4643-92df-fd1744163a64/volumes" Nov 01 00:37:43.102264 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:37:43.102184 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:37:58.757896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:58.757864 4910 generic.go:356] "Generic (PLEG): container finished" podID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerID="7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637" exitCode=1 Nov 01 00:37:58.758283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:58.757942 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" event={"ID":"b4a8bae4-5f29-4776-a979-eb6d0532e820","Type":"ContainerDied","Data":"7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637"} Nov 01 00:37:58.758283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:58.758278 4910 scope.go:122] "RemoveContainer" containerID="7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637" Nov 01 00:37:58.758646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:58.758611 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:37:58.867657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:58.867565 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:37:59.765452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:59.765423 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" event={"ID":"b4a8bae4-5f29-4776-a979-eb6d0532e820","Type":"ContainerStarted","Data":"dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91"} Nov 01 00:37:59.765860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:37:59.765477 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:38:10.773059 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:38:10.773030 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 00:38:13.102233 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:38:13.102196 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:38:30.409147 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:38:30.409120 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:38:30.409797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:38:30.409120 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:38:30.419216 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:38:30.419194 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:38:30.419353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:38:30.419339 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:38:43.101715 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:38:43.101679 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:39:05.866829 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.866783 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq"] Nov 01 00:39:05.867398 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.867378 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8fb01bf6-8775-4643-92df-fd1744163a64" containerName="manager" Nov 01 00:39:05.867481 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.867400 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb01bf6-8775-4643-92df-fd1744163a64" containerName="manager" Nov 01 00:39:05.867536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.867517 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb01bf6-8775-4643-92df-fd1744163a64" containerName="manager" Nov 01 00:39:05.870841 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.870820 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.917527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917484 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.917705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917534 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.917705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.917705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917675 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.917869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917710 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.917869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917785 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft95l\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-kube-api-access-ft95l\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.917869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917840 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.918019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917926 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.918019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.917995 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:05.975762 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:05.975727 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq"] Nov 01 00:39:06.018943 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.018913 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.018952 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.018989 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019028 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019050 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019065 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019111 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019139 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft95l\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-kube-api-access-ft95l\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019395 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019410 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019567 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.019722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.019704 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.020114 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.020090 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.021485 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.021464 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.021585 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.021549 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.080214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.080185 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.080350 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.080333 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft95l\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-kube-api-access-ft95l\") pod \"maas-default-gateway-openshift-default-67bf986f99-fwrtq\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.184341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.184278 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:06.614251 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.614228 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq"] Nov 01 00:39:06.616598 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:39:06.616564 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d38eb6_d4fd_4410_8cc4_df393808bbf9.slice/crio-1d79a124f9203bf6ee7d36efd83f35864ae02eee7f1b00e151eef671b8f81197 WatchSource:0}: Error finding container 1d79a124f9203bf6ee7d36efd83f35864ae02eee7f1b00e151eef671b8f81197: Status 404 returned error can't find the container with id 1d79a124f9203bf6ee7d36efd83f35864ae02eee7f1b00e151eef671b8f81197 Nov 01 00:39:06.618810 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.618765 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:39:06.618896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.618834 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:39:06.618896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:06.618866 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:39:07.147521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:07.147487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" event={"ID":"42d38eb6-d4fd-4410-8cc4-df393808bbf9","Type":"ContainerStarted","Data":"be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad"} Nov 01 00:39:07.147925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:07.147537 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" event={"ID":"42d38eb6-d4fd-4410-8cc4-df393808bbf9","Type":"ContainerStarted","Data":"1d79a124f9203bf6ee7d36efd83f35864ae02eee7f1b00e151eef671b8f81197"} Nov 01 00:39:07.184951 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:07.184919 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:07.186285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:07.186262 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 00:39:07.186344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:07.186313 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 00:39:08.185092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:08.185059 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 00:39:08.185496 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:08.185109 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 00:39:09.184942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:09.184863 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 00:39:09.184942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:09.184924 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 00:39:10.184640 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:10.184602 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 00:39:10.185063 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:10.184671 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 00:39:11.185642 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:11.185595 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 00:39:11.186076 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:11.185665 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 00:39:12.184692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:12.184650 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 00:39:12.184863 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:12.184716 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 00:39:13.103684 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:39:13.103651 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:39:13.185672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:13.185619 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fwrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 01 00:39:13.185893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:13.185693 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 01 00:39:14.853878 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:14.853844 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:14.854417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:14.854240 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:14.855179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:14.855153 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:39:15.027057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:39:15.027006 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podStartSLOduration=10.026992917 podStartE2EDuration="10.026992917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:39:07.274960067 +0000 UTC m=+937.254542402" watchObservedRunningTime="2026-11-01 00:39:15.026992917 +0000 UTC m=+945.006575252" Nov 01 00:39:43.101891 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:39:43.101852 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:40:13.103389 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:40:13.103341 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:40:18.993043 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:18.993007 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp"] Nov 01 00:40:18.996396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:18.996380 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.074998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.074958 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbx5\" (UniqueName: \"kubernetes.io/projected/365ddc9f-f068-42b9-8920-b4fcca9dc67c-kube-api-access-5dbx5\") pod \"payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.075231 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.075209 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365ddc9f-f068-42b9-8920-b4fcca9dc67c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.101424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.101381 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-30fb19\"" Nov 01 00:40:19.101424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.101406 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-30fb19-dockercfg-2xv2v\"" Nov 01 00:40:19.109400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.106065 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp"] Nov 01 00:40:19.176341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.176304 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbx5\" (UniqueName: \"kubernetes.io/projected/365ddc9f-f068-42b9-8920-b4fcca9dc67c-kube-api-access-5dbx5\") pod \"payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.176526 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.176366 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365ddc9f-f068-42b9-8920-b4fcca9dc67c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.177192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.177166 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365ddc9f-f068-42b9-8920-b4fcca9dc67c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.237553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.237526 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbx5\" (UniqueName: \"kubernetes.io/projected/365ddc9f-f068-42b9-8920-b4fcca9dc67c-kube-api-access-5dbx5\") pod \"payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.309471 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.309434 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:19.682319 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:19.682293 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp"] Nov 01 00:40:19.684047 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:40:19.684013 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365ddc9f_f068_42b9_8920_b4fcca9dc67c.slice/crio-a61175c963791ae0662124a596a396114d55e5768542d8c3812505eb0581ee6f WatchSource:0}: Error finding container a61175c963791ae0662124a596a396114d55e5768542d8c3812505eb0581ee6f: Status 404 returned error can't find the container with id a61175c963791ae0662124a596a396114d55e5768542d8c3812505eb0581ee6f Nov 01 00:40:20.576412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:20.576363 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" event={"ID":"365ddc9f-f068-42b9-8920-b4fcca9dc67c","Type":"ContainerStarted","Data":"a61175c963791ae0662124a596a396114d55e5768542d8c3812505eb0581ee6f"} Nov 01 00:40:21.631169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.631136 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq"] Nov 01 00:40:21.640103 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.640075 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:21.692521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.692476 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq"] Nov 01 00:40:21.699081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.699047 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a036e7-5a71-4ff4-9a77-bc23f2249437-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:21.699259 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.699112 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2csm\" (UniqueName: \"kubernetes.io/projected/56a036e7-5a71-4ff4-9a77-bc23f2249437-kube-api-access-q2csm\") pod \"payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:21.739263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.739227 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-4143b7-dockercfg-dpvjd\"" Nov 01 00:40:21.739444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.739283 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-4143b7\"" Nov 01 00:40:21.800080 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.800043 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a036e7-5a71-4ff4-9a77-bc23f2249437-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:21.800285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.800103 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2csm\" (UniqueName: \"kubernetes.io/projected/56a036e7-5a71-4ff4-9a77-bc23f2249437-kube-api-access-q2csm\") pod \"payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:21.801033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.800958 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a036e7-5a71-4ff4-9a77-bc23f2249437-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:21.862279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.862244 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2csm\" (UniqueName: \"kubernetes.io/projected/56a036e7-5a71-4ff4-9a77-bc23f2249437-kube-api-access-q2csm\") pod \"payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:21.956046 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:21.955956 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:22.351856 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.348833 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq"] Nov 01 00:40:22.358210 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:40:22.358179 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a036e7_5a71_4ff4_9a77_bc23f2249437.slice/crio-a05aaccaa68cfe4f0b6e7aff3fc17a2ecc3b8a835e01a6c775312d6eb26ed05f WatchSource:0}: Error finding container a05aaccaa68cfe4f0b6e7aff3fc17a2ecc3b8a835e01a6c775312d6eb26ed05f: Status 404 returned error can't find the container with id a05aaccaa68cfe4f0b6e7aff3fc17a2ecc3b8a835e01a6c775312d6eb26ed05f Nov 01 00:40:22.603349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.603248 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" event={"ID":"56a036e7-5a71-4ff4-9a77-bc23f2249437","Type":"ContainerStarted","Data":"cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603"} Nov 01 00:40:22.603349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.603314 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" event={"ID":"56a036e7-5a71-4ff4-9a77-bc23f2249437","Type":"ContainerStarted","Data":"a05aaccaa68cfe4f0b6e7aff3fc17a2ecc3b8a835e01a6c775312d6eb26ed05f"} Nov 01 00:40:22.603579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.603383 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:22.605019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.604985 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" event={"ID":"365ddc9f-f068-42b9-8920-b4fcca9dc67c","Type":"ContainerStarted","Data":"73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8"} Nov 01 00:40:22.605175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.605118 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:22.719851 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.719805 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" podStartSLOduration=1.719791851 podStartE2EDuration="1.719791851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:22.718660113 +0000 UTC m=+1012.698242446" watchObservedRunningTime="2026-11-01 00:40:22.719791851 +0000 UTC m=+1012.699374186" Nov 01 00:40:22.836727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:22.836677 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" podStartSLOduration=2.181328075 podStartE2EDuration="4.83666355s" totalImagesPullingTime="2.655335475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:22.835530908 +0000 UTC m=+1012.815113243" watchObservedRunningTime="2026-11-01 00:40:22.83666355 +0000 UTC m=+1012.816245885" Nov 01 00:40:26.488932 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.488897 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz"] Nov 01 00:40:26.492519 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.492504 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:26.537113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.537080 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa22de41-ac1f-4749-bf78-635a25a09efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:26.537252 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.537118 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxn2x\" (UniqueName: \"kubernetes.io/projected/aa22de41-ac1f-4749-bf78-635a25a09efc-kube-api-access-nxn2x\") pod \"payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:26.545261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.545232 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5a9c19\"" Nov 01 00:40:26.595129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.595095 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5a9c19-dockercfg-mjgf6\"" Nov 01 00:40:26.598219 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.598192 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz"] Nov 01 00:40:26.638598 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.638570 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa22de41-ac1f-4749-bf78-635a25a09efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:26.638786 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.638603 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxn2x\" (UniqueName: \"kubernetes.io/projected/aa22de41-ac1f-4749-bf78-635a25a09efc-kube-api-access-nxn2x\") pod \"payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:26.639334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.639315 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa22de41-ac1f-4749-bf78-635a25a09efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:26.701745 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.701714 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxn2x\" (UniqueName: \"kubernetes.io/projected/aa22de41-ac1f-4749-bf78-635a25a09efc-kube-api-access-nxn2x\") pod \"payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:26.806616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:26.806585 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:27.184599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:27.184569 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz"] Nov 01 00:40:27.186001 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:40:27.185971 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa22de41_ac1f_4749_bf78_635a25a09efc.slice/crio-0e8b9354b006eb21b61256f87fa59d19ede31817563dc469517b93ba72e9ce86 WatchSource:0}: Error finding container 0e8b9354b006eb21b61256f87fa59d19ede31817563dc469517b93ba72e9ce86: Status 404 returned error can't find the container with id 0e8b9354b006eb21b61256f87fa59d19ede31817563dc469517b93ba72e9ce86 Nov 01 00:40:27.640065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:27.640022 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" event={"ID":"aa22de41-ac1f-4749-bf78-635a25a09efc","Type":"ContainerStarted","Data":"3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c"} Nov 01 00:40:27.640515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:27.640123 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" event={"ID":"aa22de41-ac1f-4749-bf78-635a25a09efc","Type":"ContainerStarted","Data":"0e8b9354b006eb21b61256f87fa59d19ede31817563dc469517b93ba72e9ce86"} Nov 01 00:40:27.640515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:27.640155 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:27.758097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:27.758040 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" podStartSLOduration=1.758025211 podStartE2EDuration="1.758025211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:27.755744808 +0000 UTC m=+1017.735327142" watchObservedRunningTime="2026-11-01 00:40:27.758025211 +0000 UTC m=+1017.737607598" Nov 01 00:40:28.958498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:28.958466 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8"] Nov 01 00:40:28.962175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:28.962157 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.062526 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.062487 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0972708d-bace-41a7-a8f3-72d81578aeb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.062738 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.062571 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp422\" (UniqueName: \"kubernetes.io/projected/0972708d-bace-41a7-a8f3-72d81578aeb2-kube-api-access-tp422\") pod \"payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.066990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.066955 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-32439f-dockercfg-z8njx\"" Nov 01 00:40:29.066990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.066970 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-32439f\"" Nov 01 00:40:29.069811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.069781 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8"] Nov 01 00:40:29.112372 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.112344 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222"] Nov 01 00:40:29.115939 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.115925 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.164054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.164030 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86075eb8-bef0-4de5-b920-a7d36dff0f3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.164173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.164057 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfkt\" (UniqueName: \"kubernetes.io/projected/86075eb8-bef0-4de5-b920-a7d36dff0f3c-kube-api-access-ptfkt\") pod \"payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.164173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.164095 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0972708d-bace-41a7-a8f3-72d81578aeb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.164337 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.164308 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp422\" (UniqueName: \"kubernetes.io/projected/0972708d-bace-41a7-a8f3-72d81578aeb2-kube-api-access-tp422\") pod \"payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.164847 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.164826 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0972708d-bace-41a7-a8f3-72d81578aeb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.175244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.175218 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222"] Nov 01 00:40:29.226435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.226367 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp422\" (UniqueName: \"kubernetes.io/projected/0972708d-bace-41a7-a8f3-72d81578aeb2-kube-api-access-tp422\") pod \"payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.265238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.265199 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86075eb8-bef0-4de5-b920-a7d36dff0f3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.265238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.265229 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfkt\" (UniqueName: \"kubernetes.io/projected/86075eb8-bef0-4de5-b920-a7d36dff0f3c-kube-api-access-ptfkt\") pod \"payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.266123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.266095 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86075eb8-bef0-4de5-b920-a7d36dff0f3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.277102 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.277078 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:29.329513 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.329478 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfkt\" (UniqueName: \"kubernetes.io/projected/86075eb8-bef0-4de5-b920-a7d36dff0f3c-kube-api-access-ptfkt\") pod \"payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.430082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.430050 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:29.661613 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.661590 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8"] Nov 01 00:40:29.663084 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:40:29.663063 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0972708d_bace_41a7_a8f3_72d81578aeb2.slice/crio-5976d275c8e5299c74a8560a2d36079b748d9a94723b99487777358a48bf7057 WatchSource:0}: Error finding container 5976d275c8e5299c74a8560a2d36079b748d9a94723b99487777358a48bf7057: Status 404 returned error can't find the container with id 5976d275c8e5299c74a8560a2d36079b748d9a94723b99487777358a48bf7057 Nov 01 00:40:29.817502 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:29.817472 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222"] Nov 01 00:40:29.818795 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:40:29.818767 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86075eb8_bef0_4de5_b920_a7d36dff0f3c.slice/crio-16b6b1e2496631eac88909113b2713bef042ccd3157db596d00e620690e7e9c9 WatchSource:0}: Error finding container 16b6b1e2496631eac88909113b2713bef042ccd3157db596d00e620690e7e9c9: Status 404 returned error can't find the container with id 16b6b1e2496631eac88909113b2713bef042ccd3157db596d00e620690e7e9c9 Nov 01 00:40:30.660325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.660285 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" event={"ID":"0972708d-bace-41a7-a8f3-72d81578aeb2","Type":"ContainerStarted","Data":"408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7"} Nov 01 00:40:30.660866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.660353 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" event={"ID":"0972708d-bace-41a7-a8f3-72d81578aeb2","Type":"ContainerStarted","Data":"5976d275c8e5299c74a8560a2d36079b748d9a94723b99487777358a48bf7057"} Nov 01 00:40:30.660866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.660380 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:30.662234 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.662206 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" event={"ID":"86075eb8-bef0-4de5-b920-a7d36dff0f3c","Type":"ContainerStarted","Data":"97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced"} Nov 01 00:40:30.662353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.662252 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" event={"ID":"86075eb8-bef0-4de5-b920-a7d36dff0f3c","Type":"ContainerStarted","Data":"16b6b1e2496631eac88909113b2713bef042ccd3157db596d00e620690e7e9c9"} Nov 01 00:40:30.662353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.662303 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:30.778072 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.777927 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" podStartSLOduration=2.77791304 podStartE2EDuration="2.77791304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:30.775197459 +0000 UTC m=+1020.754779793" watchObservedRunningTime="2026-11-01 00:40:30.77791304 +0000 UTC m=+1020.757495369" Nov 01 00:40:30.895562 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:30.895507 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" podStartSLOduration=1.895492463 podStartE2EDuration="1.895492463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:30.893330159 +0000 UTC m=+1020.872912494" watchObservedRunningTime="2026-11-01 00:40:30.895492463 +0000 UTC m=+1020.875074815" Nov 01 00:40:33.613428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:33.613403 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 00:40:33.613860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:33.613464 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 00:40:36.450886 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.450840 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44"] Nov 01 00:40:36.456475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.456448 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.509481 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.509451 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-43a0cf\"" Nov 01 00:40:36.511276 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.511250 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44"] Nov 01 00:40:36.537044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.537013 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88ml\" (UniqueName: \"kubernetes.io/projected/1b98851f-c830-4d22-a98b-8e1de94bde25-kube-api-access-h88ml\") pod \"payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.537175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.537086 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98851f-c830-4d22-a98b-8e1de94bde25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.588670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.588642 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d"] Nov 01 00:40:36.592263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.592246 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:36.638005 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.637975 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98851f-c830-4d22-a98b-8e1de94bde25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.638097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.638037 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:36.638146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.638123 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqllt\" (UniqueName: \"kubernetes.io/projected/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-kube-api-access-wqllt\") pod \"payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:36.638179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.638156 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88ml\" (UniqueName: \"kubernetes.io/projected/1b98851f-c830-4d22-a98b-8e1de94bde25-kube-api-access-h88ml\") pod \"payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.638722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.638707 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98851f-c830-4d22-a98b-8e1de94bde25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.648196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.648147 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-43a0cf-dockercfg-2998n\"" Nov 01 00:40:36.653036 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.653013 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d"] Nov 01 00:40:36.699726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.699691 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88ml\" (UniqueName: \"kubernetes.io/projected/1b98851f-c830-4d22-a98b-8e1de94bde25-kube-api-access-h88ml\") pod \"payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.739193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.739121 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:36.739193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.739169 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqllt\" (UniqueName: \"kubernetes.io/projected/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-kube-api-access-wqllt\") pod \"payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:36.739909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.739888 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:36.775675 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.775647 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:36.800226 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.800203 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqllt\" (UniqueName: \"kubernetes.io/projected/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-kube-api-access-wqllt\") pod \"payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:36.905262 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:36.905234 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:37.152961 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.152935 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44"] Nov 01 00:40:37.154293 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:40:37.154265 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b98851f_c830_4d22_a98b_8e1de94bde25.slice/crio-51ce024bfcda0facc8b1a74ef0483724d0b8d07596827317fa847b4bb4e46f42 WatchSource:0}: Error finding container 51ce024bfcda0facc8b1a74ef0483724d0b8d07596827317fa847b4bb4e46f42: Status 404 returned error can't find the container with id 51ce024bfcda0facc8b1a74ef0483724d0b8d07596827317fa847b4bb4e46f42 Nov 01 00:40:37.280451 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.280428 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d"] Nov 01 00:40:37.281422 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:40:37.281396 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4280bb5b_9384_4b89_93b1_f8e5bb62ceb4.slice/crio-2b7accce5d36318dd4a08e694f567b448fa5aa1249aee3dd073ca81a13da2fb5 WatchSource:0}: Error finding container 2b7accce5d36318dd4a08e694f567b448fa5aa1249aee3dd073ca81a13da2fb5: Status 404 returned error can't find the container with id 2b7accce5d36318dd4a08e694f567b448fa5aa1249aee3dd073ca81a13da2fb5 Nov 01 00:40:37.708702 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.708660 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" event={"ID":"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4","Type":"ContainerStarted","Data":"91df9370a96e053d7b0a5ee094790b292c5bc505bec97eae3a4f448804ebac04"} Nov 01 00:40:37.709149 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.708726 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:40:37.709149 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.708749 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" event={"ID":"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4","Type":"ContainerStarted","Data":"2b7accce5d36318dd4a08e694f567b448fa5aa1249aee3dd073ca81a13da2fb5"} Nov 01 00:40:37.711389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.710399 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" event={"ID":"1b98851f-c830-4d22-a98b-8e1de94bde25","Type":"ContainerStarted","Data":"6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b"} Nov 01 00:40:37.711389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.710461 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:37.711389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.710487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" event={"ID":"1b98851f-c830-4d22-a98b-8e1de94bde25","Type":"ContainerStarted","Data":"51ce024bfcda0facc8b1a74ef0483724d0b8d07596827317fa847b4bb4e46f42"} Nov 01 00:40:37.825682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.825608 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" podStartSLOduration=1.82559309 podStartE2EDuration="1.82559309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:37.824262206 +0000 UTC m=+1027.803844542" watchObservedRunningTime="2026-11-01 00:40:37.82559309 +0000 UTC m=+1027.805175424" Nov 01 00:40:37.944278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:37.944203 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" podStartSLOduration=1.944185589 podStartE2EDuration="1.944185589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:40:37.939817213 +0000 UTC m=+1027.919399551" watchObservedRunningTime="2026-11-01 00:40:37.944185589 +0000 UTC m=+1027.923767925" Nov 01 00:40:38.650752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:38.650726 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 00:40:41.673618 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:41.673591 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 00:40:41.674050 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:41.673682 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 00:40:43.101298 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:40:43.101263 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:40:48.720010 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:48.719982 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 00:40:48.720428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:40:48.720059 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 00:41:06.017434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.017397 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6"] Nov 01 00:41:06.024905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.024883 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.080483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.080453 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6"] Nov 01 00:41:06.113555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.113514 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-dshm\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.113727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.113599 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-tmp-dir\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.113727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.113712 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-home\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.113814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.113731 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-model-cache\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.113848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.113814 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2895da5-d473-4c4b-8288-8a055f3fb483-tls-certs\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.113881 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.113848 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m829z\" (UniqueName: \"kubernetes.io/projected/f2895da5-d473-4c4b-8288-8a055f3fb483-kube-api-access-m829z\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.128146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.128118 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"kube-root-ca.crt\"" Nov 01 00:41:06.128284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.128268 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"openshift-service-ca.crt\"" Nov 01 00:41:06.128327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.128275 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"default-dockercfg-xn6ld\"" Nov 01 00:41:06.128718 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.128704 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"simulator-w6-22d7b2-kserve-self-signed-certs\"" Nov 01 00:41:06.215361 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.215292 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-tmp-dir\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.215569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.215389 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-home\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.215569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.215429 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-model-cache\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.215569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.215467 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2895da5-d473-4c4b-8288-8a055f3fb483-tls-certs\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.215569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.215494 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m829z\" (UniqueName: \"kubernetes.io/projected/f2895da5-d473-4c4b-8288-8a055f3fb483-kube-api-access-m829z\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.215569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.215535 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-dshm\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.216399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.216377 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-model-cache\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.216571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.216387 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-home\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.216748 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.216727 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-tmp-dir\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.219068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.219043 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-dshm\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.219305 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.219285 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2895da5-d473-4c4b-8288-8a055f3fb483-tls-certs\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.278669 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.278586 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m829z\" (UniqueName: \"kubernetes.io/projected/f2895da5-d473-4c4b-8288-8a055f3fb483-kube-api-access-m829z\") pod \"simulator-w6-22d7b2-kserve-59cccf57cd-b57j6\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.337704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.337671 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:06.733438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.733410 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6"] Nov 01 00:41:06.734841 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:41:06.734815 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2895da5_d473_4c4b_8288_8a055f3fb483.slice/crio-a1ff0d4042e2e3ae951f9f9c85a422e92b12ee2061e589d61548fbda76e98ba0 WatchSource:0}: Error finding container a1ff0d4042e2e3ae951f9f9c85a422e92b12ee2061e589d61548fbda76e98ba0: Status 404 returned error can't find the container with id a1ff0d4042e2e3ae951f9f9c85a422e92b12ee2061e589d61548fbda76e98ba0 Nov 01 00:41:06.888432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:06.888395 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" event={"ID":"f2895da5-d473-4c4b-8288-8a055f3fb483","Type":"ContainerStarted","Data":"a1ff0d4042e2e3ae951f9f9c85a422e92b12ee2061e589d61548fbda76e98ba0"} Nov 01 00:41:09.915604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:09.915571 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" event={"ID":"f2895da5-d473-4c4b-8288-8a055f3fb483","Type":"ContainerStarted","Data":"da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c"} Nov 01 00:41:10.049163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:10.049098 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podStartSLOduration=2.566597544 podStartE2EDuration="5.049080832s" totalImagesPullingTime="2.482483288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:41:10.047891447 +0000 UTC m=+1060.027473789" watchObservedRunningTime="2026-11-01 00:41:10.049080832 +0000 UTC m=+1060.028663170" Nov 01 00:41:10.337951 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:10.337921 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:13.108288 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:41:13.108247 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:41:16.338121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:16.338089 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:16.350857 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:16.350831 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:16.969810 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:16.969786 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:41:43.104567 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:41:43.104532 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:41:51.032324 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.032287 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt"] Nov 01 00:41:51.036032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.036016 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.130281 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.130248 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-tmp-dir\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.130457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.130291 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-home\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.130457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.130310 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-model-cache\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.130457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.130404 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866123ba-31f1-4403-bdd1-53a699b2d4f9-tls-certs\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.130457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.130444 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmgs\" (UniqueName: \"kubernetes.io/projected/866123ba-31f1-4403-bdd1-53a699b2d4f9-kube-api-access-9gmgs\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.130726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.130592 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-dshm\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.141993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.141973 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"default-dockercfg-n9ktj\"" Nov 01 00:41:51.142089 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.142057 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"simulator-w3-5a9c19-kserve-self-signed-certs\"" Nov 01 00:41:51.142144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.142111 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"kube-root-ca.crt\"" Nov 01 00:41:51.142144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.142118 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5a9c19\"/\"openshift-service-ca.crt\"" Nov 01 00:41:51.144796 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.144776 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt"] Nov 01 00:41:51.232141 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232105 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-tmp-dir\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232153 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-home\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232259 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-model-cache\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232303 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866123ba-31f1-4403-bdd1-53a699b2d4f9-tls-certs\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232466 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232320 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmgs\" (UniqueName: \"kubernetes.io/projected/866123ba-31f1-4403-bdd1-53a699b2d4f9-kube-api-access-9gmgs\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232466 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232386 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-dshm\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232574 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232543 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-home\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232668 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232577 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-tmp-dir\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.232745 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.232690 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-model-cache\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.234790 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.234769 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-dshm\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.235931 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.235911 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866123ba-31f1-4403-bdd1-53a699b2d4f9-tls-certs\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.294450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.294372 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmgs\" (UniqueName: \"kubernetes.io/projected/866123ba-31f1-4403-bdd1-53a699b2d4f9-kube-api-access-9gmgs\") pod \"simulator-w3-5a9c19-kserve-699dfbd894-87mzt\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.348186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.348128 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:41:51.745544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:51.745521 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt"] Nov 01 00:41:51.746457 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:41:51.746431 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod866123ba_31f1_4403_bdd1_53a699b2d4f9.slice/crio-9754cc2afe0228ef8934fad4154479d514ed04a4866b1e07e71c97801da8eeab WatchSource:0}: Error finding container 9754cc2afe0228ef8934fad4154479d514ed04a4866b1e07e71c97801da8eeab: Status 404 returned error can't find the container with id 9754cc2afe0228ef8934fad4154479d514ed04a4866b1e07e71c97801da8eeab Nov 01 00:41:52.178245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:52.178210 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" event={"ID":"866123ba-31f1-4403-bdd1-53a699b2d4f9","Type":"ContainerStarted","Data":"801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490"} Nov 01 00:41:52.178688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:52.178262 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" event={"ID":"866123ba-31f1-4403-bdd1-53a699b2d4f9","Type":"ContainerStarted","Data":"9754cc2afe0228ef8934fad4154479d514ed04a4866b1e07e71c97801da8eeab"} Nov 01 00:41:52.297558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:52.297499 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podStartSLOduration=2.297484142 podStartE2EDuration="2.297484142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:41:52.295659125 +0000 UTC m=+1102.275241462" watchObservedRunningTime="2026-11-01 00:41:52.297484142 +0000 UTC m=+1102.277066476" Nov 01 00:41:52.349147 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:41:52.349118 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:42:01.349029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:01.348987 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:42:01.362642 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:01.362597 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:42:02.252270 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:02.252232 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:42:13.103085 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:42:13.103046 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:42:16.524775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.524742 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl"] Nov 01 00:42:16.528544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.528525 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.579534 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.579504 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-model-cache\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.579726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.579549 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tls-certs\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.579726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.579709 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-dshm\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.579803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.579747 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8n6\" (UniqueName: \"kubernetes.io/projected/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-kube-api-access-db8n6\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.579803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.579767 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-home\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.579868 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.579834 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tmp-dir\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.582340 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.582325 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"simulator-w5-948e22-kserve-self-signed-certs\"" Nov 01 00:42:16.632313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.632278 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"openshift-service-ca.crt\"" Nov 01 00:42:16.632313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.632297 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"kube-root-ca.crt\"" Nov 01 00:42:16.635182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.635158 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl"] Nov 01 00:42:16.680913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.680877 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-dshm\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.680924 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8n6\" (UniqueName: \"kubernetes.io/projected/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-kube-api-access-db8n6\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.680951 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-home\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.680980 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tmp-dir\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.681023 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-model-cache\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.681068 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tls-certs\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681472 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.681433 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-home\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681584 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.681474 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tmp-dir\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.681584 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.681441 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-model-cache\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.683469 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.683444 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-dshm\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.683738 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.683719 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tls-certs\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.690779 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.690761 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-948e22\"/\"default-dockercfg-xqcln\"" Nov 01 00:42:16.745402 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.745366 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8n6\" (UniqueName: \"kubernetes.io/projected/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-kube-api-access-db8n6\") pod \"simulator-w5-948e22-kserve-b8dfbd98b-c7hgl\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:16.842912 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:16.842823 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:17.234499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:17.234474 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl"] Nov 01 00:42:17.236188 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:42:17.236166 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50e2a7d3_54ff_4745_8d7e_2ba23a1a7326.slice/crio-e4bb04893e42432758aae7f46f58aaf20104ada3414e9ef444544b116ad91395 WatchSource:0}: Error finding container e4bb04893e42432758aae7f46f58aaf20104ada3414e9ef444544b116ad91395: Status 404 returned error can't find the container with id e4bb04893e42432758aae7f46f58aaf20104ada3414e9ef444544b116ad91395 Nov 01 00:42:17.336190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:17.336156 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" event={"ID":"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326","Type":"ContainerStarted","Data":"67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1"} Nov 01 00:42:17.336190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:17.336193 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" event={"ID":"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326","Type":"ContainerStarted","Data":"e4bb04893e42432758aae7f46f58aaf20104ada3414e9ef444544b116ad91395"} Nov 01 00:42:17.473456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:17.473404 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podStartSLOduration=1.47338606 podStartE2EDuration="1.47338606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:42:17.470836547 +0000 UTC m=+1127.450418885" watchObservedRunningTime="2026-11-01 00:42:17.47338606 +0000 UTC m=+1127.452968395" Nov 01 00:42:17.843587 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:17.843548 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:26.843597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:26.843562 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:26.856284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:26.856262 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:27.412294 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:27.412263 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:42:43.103824 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:42:43.103789 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:42:57.305092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.305054 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5"] Nov 01 00:42:57.308924 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.308908 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.368724 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.368691 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5"] Nov 01 00:42:57.371068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.371040 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-model-cache\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.371200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.371080 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46be65a6-4f18-4e60-855c-6b08571e2981-tls-certs\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.371261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.371189 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-dshm\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.371261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.371228 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-home\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.371394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.371369 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kdp\" (UniqueName: \"kubernetes.io/projected/46be65a6-4f18-4e60-855c-6b08571e2981-kube-api-access-s2kdp\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.371449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.371416 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-tmp-dir\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.414389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.414361 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"openshift-service-ca.crt\"" Nov 01 00:42:57.414555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.414538 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"kube-root-ca.crt\"" Nov 01 00:42:57.414602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.414539 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"default-dockercfg-l7qsj\"" Nov 01 00:42:57.414668 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.414652 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"premium-w4-43a0cf-kserve-self-signed-certs\"" Nov 01 00:42:57.471975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.471942 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-home\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472264 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472008 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kdp\" (UniqueName: \"kubernetes.io/projected/46be65a6-4f18-4e60-855c-6b08571e2981-kube-api-access-s2kdp\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472264 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472040 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-tmp-dir\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472264 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472103 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-model-cache\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472264 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472147 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46be65a6-4f18-4e60-855c-6b08571e2981-tls-certs\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472264 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472193 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-dshm\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472422 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-home\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472457 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-tmp-dir\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.472536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.472493 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-model-cache\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.474569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.474538 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-dshm\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.474925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.474907 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46be65a6-4f18-4e60-855c-6b08571e2981-tls-certs\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.533451 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.533432 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kdp\" (UniqueName: \"kubernetes.io/projected/46be65a6-4f18-4e60-855c-6b08571e2981-kube-api-access-s2kdp\") pod \"premium-w4-43a0cf-kserve-74457c7b9c-rcqb5\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:57.620614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:57.620527 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:58.027555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:58.027526 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5"] Nov 01 00:42:58.028871 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:42:58.028837 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46be65a6_4f18_4e60_855c_6b08571e2981.slice/crio-65f76930b2a7463ad50fef90dd69b039f6cc821756bfadb462f108992c55f563 WatchSource:0}: Error finding container 65f76930b2a7463ad50fef90dd69b039f6cc821756bfadb462f108992c55f563: Status 404 returned error can't find the container with id 65f76930b2a7463ad50fef90dd69b039f6cc821756bfadb462f108992c55f563 Nov 01 00:42:58.597085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:58.597048 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" event={"ID":"46be65a6-4f18-4e60-855c-6b08571e2981","Type":"ContainerStarted","Data":"eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a"} Nov 01 00:42:58.597085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:58.597093 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" event={"ID":"46be65a6-4f18-4e60-855c-6b08571e2981","Type":"ContainerStarted","Data":"65f76930b2a7463ad50fef90dd69b039f6cc821756bfadb462f108992c55f563"} Nov 01 00:42:58.621589 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:58.621565 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:42:58.715432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:42:58.715371 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podStartSLOduration=1.715356915 podStartE2EDuration="1.715356915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:42:58.713584523 +0000 UTC m=+1168.693166861" watchObservedRunningTime="2026-11-01 00:42:58.715356915 +0000 UTC m=+1168.694939276" Nov 01 00:43:07.621447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:07.621408 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:43:07.634186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:07.634161 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:43:07.664640 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:07.664602 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:43:13.102656 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:43:13.102605 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:43:30.462780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:30.462751 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:43:30.463349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:30.463169 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:43:30.474132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:30.474104 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:43:30.474265 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:30.474171 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:43:34.349498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.349469 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5"] Nov 01 00:43:34.353603 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.353581 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.406359 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.406325 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-43a0cf-openshift-default-dockercfg-9jwvj\"" Nov 01 00:43:34.460083 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.460053 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5"] Nov 01 00:43:34.534290 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534261 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534305 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534333 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534506 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534467 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85m4\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-kube-api-access-k85m4\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534521 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534557 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534611 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d85d550-af31-494e-aee4-d30f9b0900e5-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534737 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534725 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.534775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.534747 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635649 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85m4\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-kube-api-access-k85m4\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635698 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635724 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635960 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635739 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d85d550-af31-494e-aee4-d30f9b0900e5-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635960 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635769 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635960 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635796 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635960 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635859 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.635960 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.635911 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.636182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.636050 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.636275 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.636247 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.636412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.636294 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.636412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.636323 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.636517 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.636429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.636681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.636663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d85d550-af31-494e-aee4-d30f9b0900e5-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.638332 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.638311 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.638664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.638645 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.697279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.697251 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.697402 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.697290 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85m4\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-kube-api-access-k85m4\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:34.968493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:34.968404 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:35.406925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.406898 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5"] Nov 01 00:43:35.408499 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:43:35.408472 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d85d550_af31_494e_aee4_d30f9b0900e5.slice/crio-309bd62be4e00984a10536facc400339206f9098d8cbe495d92703be1ec3d73a WatchSource:0}: Error finding container 309bd62be4e00984a10536facc400339206f9098d8cbe495d92703be1ec3d73a: Status 404 returned error can't find the container with id 309bd62be4e00984a10536facc400339206f9098d8cbe495d92703be1ec3d73a Nov 01 00:43:35.410708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.410679 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:43:35.410797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.410740 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:43:35.410797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.410769 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:43:35.840142 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.840107 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" event={"ID":"4d85d550-af31-494e-aee4-d30f9b0900e5","Type":"ContainerStarted","Data":"e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0"} Nov 01 00:43:35.840332 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.840162 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" event={"ID":"4d85d550-af31-494e-aee4-d30f9b0900e5","Type":"ContainerStarted","Data":"309bd62be4e00984a10536facc400339206f9098d8cbe495d92703be1ec3d73a"} Nov 01 00:43:35.975651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.974735 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:35.982223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:35.982158 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" podStartSLOduration=1.98213868 podStartE2EDuration="1.98213868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:43:35.981399208 +0000 UTC m=+1205.960981543" watchObservedRunningTime="2026-11-01 00:43:35.98213868 +0000 UTC m=+1205.961721014" Nov 01 00:43:36.972649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:36.972396 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 00:43:36.972649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:36.972455 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 00:43:37.972651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:37.971994 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 00:43:37.972651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:37.972056 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 00:43:38.975660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:38.974798 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 00:43:38.975660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:38.974902 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 00:43:39.974023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:39.973987 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 00:43:39.974306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:39.974277 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 00:43:40.973483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:40.972979 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 01 00:43:40.973483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:40.973062 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 01 00:43:42.108739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:42.108692 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:42.109276 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:42.109256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:42.110257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:42.110228 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 00:43:43.104044 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:43:43.104010 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:43:46.213320 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.213289 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-6b79g-s2nxb"] Nov 01 00:43:46.217016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.217001 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.267369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.267343 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-service-ca-bundle\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.267503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.267463 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-trusted-ca-bundle\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.267503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.267492 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/14899c7c-f079-4e5a-b044-9cc72f305b83-archives-path\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.267650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.267593 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cq2\" (UniqueName: \"kubernetes.io/projected/14899c7c-f079-4e5a-b044-9cc72f305b83-kube-api-access-c5cq2\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.273955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.273932 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6b79g-s2nxb"] Nov 01 00:43:46.368154 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.368127 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-trusted-ca-bundle\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.368315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.368162 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/14899c7c-f079-4e5a-b044-9cc72f305b83-archives-path\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.368315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.368196 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cq2\" (UniqueName: \"kubernetes.io/projected/14899c7c-f079-4e5a-b044-9cc72f305b83-kube-api-access-c5cq2\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.368315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.368246 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-service-ca-bundle\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.368593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.368571 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/14899c7c-f079-4e5a-b044-9cc72f305b83-archives-path\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.368891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.368874 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-service-ca-bundle\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.369347 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.369327 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-trusted-ca-bundle\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.433123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.433095 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cq2\" (UniqueName: \"kubernetes.io/projected/14899c7c-f079-4e5a-b044-9cc72f305b83-kube-api-access-c5cq2\") pod \"periodic-gathering-6b79g-s2nxb\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.529515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.529495 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:43:46.904051 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.904019 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6b79g-s2nxb"] Nov 01 00:43:46.905352 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:43:46.905322 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14899c7c_f079_4e5a_b044_9cc72f305b83.slice/crio-0f45f8a5892220ec89f199e0c191071bc0edbec230364d2f0c855469c2584b93 WatchSource:0}: Error finding container 0f45f8a5892220ec89f199e0c191071bc0edbec230364d2f0c855469c2584b93: Status 404 returned error can't find the container with id 0f45f8a5892220ec89f199e0c191071bc0edbec230364d2f0c855469c2584b93 Nov 01 00:43:46.935697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:46.935667 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" event={"ID":"14899c7c-f079-4e5a-b044-9cc72f305b83","Type":"ContainerStarted","Data":"0f45f8a5892220ec89f199e0c191071bc0edbec230364d2f0c855469c2584b93"} Nov 01 00:43:47.945795 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:47.945751 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" event={"ID":"14899c7c-f079-4e5a-b044-9cc72f305b83","Type":"ContainerStarted","Data":"6e311a980737a5f7b9f0777e7c795fe3f33e9f0b7cf27d81b2fdac89e69c98b1"} Nov 01 00:43:59.430635 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.430567 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" podStartSLOduration=13.430552151 podStartE2EDuration="13.430552151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:43:48.064715125 +0000 UTC m=+1218.044297460" watchObservedRunningTime="2026-11-01 00:43:59.430552151 +0000 UTC m=+1229.410134527" Nov 01 00:43:59.431746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.431714 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp"] Nov 01 00:43:59.435755 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.435734 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.488976 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.488955 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:43:59.491336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.491296 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:43:59.500923 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.500894 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.501029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.500936 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.501029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.500955 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869q5\" (UniqueName: \"kubernetes.io/projected/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-kube-api-access-869q5\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.542174 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.542149 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp"] Nov 01 00:43:59.602160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.602130 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.602378 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.602359 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.602498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.602485 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869q5\" (UniqueName: \"kubernetes.io/projected/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-kube-api-access-869q5\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.604690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.604368 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.610011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.609989 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.664400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.664374 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-869q5\" (UniqueName: \"kubernetes.io/projected/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-kube-api-access-869q5\") pod \"maas-api-revoke-sub-5e630b1a-cl6zp\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:43:59.751959 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:43:59.751936 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:44:00.133765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:00.133727 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp"] Nov 01 00:44:00.134383 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:00.134356 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d316eaf_900e_48d1_8a6d_5b35d81bb1b3.slice/crio-0ba5f4b586fd9d7c902d6a4ffef4b956d34ec973f435709fbfd9eb4a11d4354b WatchSource:0}: Error finding container 0ba5f4b586fd9d7c902d6a4ffef4b956d34ec973f435709fbfd9eb4a11d4354b: Status 404 returned error can't find the container with id 0ba5f4b586fd9d7c902d6a4ffef4b956d34ec973f435709fbfd9eb4a11d4354b Nov 01 00:44:00.136838 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:00.136818 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:44:01.041860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:01.041823 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" event={"ID":"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3","Type":"ContainerStarted","Data":"0ba5f4b586fd9d7c902d6a4ffef4b956d34ec973f435709fbfd9eb4a11d4354b"} Nov 01 00:44:06.083193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:06.083147 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" event={"ID":"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3","Type":"ContainerStarted","Data":"e08a8bc5e20cbd406166db254fbf0cdb50039ceaaeb4c05e695400fcee8416a0"} Nov 01 00:44:06.202114 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:06.202053 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" podStartSLOduration=1.935558243 podStartE2EDuration="7.202035967s" totalImagesPullingTime="5.266477724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:44:06.200268499 +0000 UTC m=+1236.179850835" watchObservedRunningTime="2026-11-01 00:44:06.202035967 +0000 UTC m=+1236.181618304" Nov 01 00:44:07.090696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:07.090658 4910 generic.go:356] "Generic (PLEG): container finished" podID="1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" containerID="e08a8bc5e20cbd406166db254fbf0cdb50039ceaaeb4c05e695400fcee8416a0" exitCode=0 Nov 01 00:44:07.091155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:07.090768 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" event={"ID":"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3","Type":"ContainerDied","Data":"e08a8bc5e20cbd406166db254fbf0cdb50039ceaaeb4c05e695400fcee8416a0"} Nov 01 00:44:08.216684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.216615 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq"] Nov 01 00:44:08.217051 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.216890 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" containerID="cri-o://be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad" gracePeriod=30 Nov 01 00:44:08.238528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.238505 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:44:08.394919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.394846 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-kube-api-access-869q5\" (UniqueName: \"kubernetes.io/projected/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-kube-api-access-869q5\") pod \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " Nov 01 00:44:08.395065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.394925 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-openshift-ca\") pod \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " Nov 01 00:44:08.395065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.394957 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-xks-ca\") pod \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\" (UID: \"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3\") " Nov 01 00:44:08.395716 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.395580 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-openshift-ca" pod "1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" (UID: "1d316eaf-900e-48d1-8a6d-5b35d81bb1b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:08.397796 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.397766 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-kube-api-access-869q5" pod "1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" (UID: "1d316eaf-900e-48d1-8a6d-5b35d81bb1b3"). InnerVolumeSpecName "kube-api-access-869q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:08.398304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.398278 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-xks-ca" pod "1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" (UID: "1d316eaf-900e-48d1-8a6d-5b35d81bb1b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:44:08.496842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.496814 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869q5\" (UniqueName: \"kubernetes.io/projected/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-kube-api-access-869q5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:08.496842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.496840 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:08.497029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:08.496852 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:09.105511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:09.105460 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" event={"ID":"1d316eaf-900e-48d1-8a6d-5b35d81bb1b3","Type":"ContainerDied","Data":"0ba5f4b586fd9d7c902d6a4ffef4b956d34ec973f435709fbfd9eb4a11d4354b"} Nov 01 00:44:09.105715 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:09.105513 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp" Nov 01 00:44:09.105715 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:09.105526 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ba5f4b586fd9d7c902d6a4ffef4b956d34ec973f435709fbfd9eb4a11d4354b" Nov 01 00:44:13.103547 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:44:13.103502 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:44:13.502604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.502576 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:44:13.650315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650243 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-kube-api-access-ft95l\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-kube-api-access-ft95l\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650282 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-socket\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650530 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650320 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-credential-socket\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650530 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650357 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-podinfo\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650530 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650426 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-data\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650530 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650447 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istiod-ca-cert\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650530 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650485 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-certs\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650531 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-token\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-token\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650567 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-envoy\") pod \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\" (UID: \"42d38eb6-d4fd-4410-8cc4-df393808bbf9\") " Nov 01 00:44:13.650803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650670 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-socket" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:44:13.650803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650690 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-credential-socket" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:44:13.650803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650717 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-data" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:44:13.651078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.650915 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istiod-ca-cert" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:13.651415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.651171 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-certs" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:44:13.651415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.651280 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.651415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.651371 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.651415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.651389 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.651415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.651404 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.652756 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.652726 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-kube-api-access-ft95l" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "kube-api-access-ft95l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:13.652870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.652812 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-podinfo" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:44:13.652870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.652856 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-envoy" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:44:13.652950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.652920 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-token" pod "42d38eb6-d4fd-4410-8cc4-df393808bbf9" (UID: "42d38eb6-d4fd-4410-8cc4-df393808bbf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:13.752597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.752566 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.752597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.752593 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.752597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.752604 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.752866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.752613 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft95l\" (UniqueName: \"kubernetes.io/projected/42d38eb6-d4fd-4410-8cc4-df393808bbf9-kube-api-access-ft95l\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:13.752866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:13.752641 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d38eb6-d4fd-4410-8cc4-df393808bbf9-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:14.151464 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.151429 4910 generic.go:356] "Generic (PLEG): container finished" podID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerID="be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad" exitCode=0 Nov 01 00:44:14.151894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.151507 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" Nov 01 00:44:14.151894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.151505 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" event={"ID":"42d38eb6-d4fd-4410-8cc4-df393808bbf9","Type":"ContainerDied","Data":"be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad"} Nov 01 00:44:14.151894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.151669 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq" event={"ID":"42d38eb6-d4fd-4410-8cc4-df393808bbf9","Type":"ContainerDied","Data":"1d79a124f9203bf6ee7d36efd83f35864ae02eee7f1b00e151eef671b8f81197"} Nov 01 00:44:14.151894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.151694 4910 scope.go:122] "RemoveContainer" containerID="be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad" Nov 01 00:44:14.162668 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.162646 4910 scope.go:122] "RemoveContainer" containerID="be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad" Nov 01 00:44:14.162938 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:44:14.162920 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad\": container with ID starting with be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad not found: ID does not exist" containerID="be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad" Nov 01 00:44:14.163036 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.162949 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad"} err="failed to get container status \"be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad\": rpc error: code = NotFound desc = could not find container \"be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad\": container with ID starting with be8cf7b0221efb34040026ff49480d36f962819dd150317c3a81871b6c2287ad not found: ID does not exist" Nov 01 00:44:14.370092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.370054 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq"] Nov 01 00:44:14.420400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.420314 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fwrtq"] Nov 01 00:44:14.976554 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.976520 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74"] Nov 01 00:44:14.977334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.977313 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" containerName="revoke-keys" Nov 01 00:44:14.977385 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.977339 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" containerName="revoke-keys" Nov 01 00:44:14.977427 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.977398 4910 cpu_manager.go:414] "containerMap: removing container" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" Nov 01 00:44:14.977427 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.977407 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" Nov 01 00:44:14.977541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.977528 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" containerName="revoke-keys" Nov 01 00:44:14.977576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.977555 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" containerName="istio-proxy" Nov 01 00:44:14.982931 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:14.982907 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.036319 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.036284 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:44:15.036525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.036396 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:44:15.040034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.040004 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74"] Nov 01 00:44:15.165021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.164991 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bdk\" (UniqueName: \"kubernetes.io/projected/174b1886-1f98-47a8-8463-b7f3539be035-kube-api-access-v2bdk\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.165021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.165023 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.165447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.165186 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.268724 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.266248 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bdk\" (UniqueName: \"kubernetes.io/projected/174b1886-1f98-47a8-8463-b7f3539be035-kube-api-access-v2bdk\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.268724 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.266301 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.268724 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.266398 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.268724 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.267774 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.275791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.275768 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.339591 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.335410 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bdk\" (UniqueName: \"kubernetes.io/projected/174b1886-1f98-47a8-8463-b7f3539be035-kube-api-access-v2bdk\") pod \"maas-api-revoke-sub-33e5cf46-2jg74\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:15.599039 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:15.598955 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:16.017583 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:16.015649 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74"] Nov 01 00:44:16.018701 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:16.018637 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174b1886_1f98_47a8_8463_b7f3539be035.slice/crio-368e3bbdbac6b57378889942b1d206e9297f972201db6f53e0e79ca68d5ae0a4 WatchSource:0}: Error finding container 368e3bbdbac6b57378889942b1d206e9297f972201db6f53e0e79ca68d5ae0a4: Status 404 returned error can't find the container with id 368e3bbdbac6b57378889942b1d206e9297f972201db6f53e0e79ca68d5ae0a4 Nov 01 00:44:16.169031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:16.168988 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" event={"ID":"174b1886-1f98-47a8-8463-b7f3539be035","Type":"ContainerStarted","Data":"368e3bbdbac6b57378889942b1d206e9297f972201db6f53e0e79ca68d5ae0a4"} Nov 01 00:44:16.206653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:16.203600 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d38eb6-d4fd-4410-8cc4-df393808bbf9" path="/var/lib/kubelet/pods/42d38eb6-d4fd-4410-8cc4-df393808bbf9/volumes" Nov 01 00:44:17.195181 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:17.195151 4910 generic.go:356] "Generic (PLEG): container finished" podID="174b1886-1f98-47a8-8463-b7f3539be035" containerID="09d0cbc31753a1809d83cae21d32437f8038eb57943ed223144cc54fed50e347" exitCode=0 Nov 01 00:44:17.195568 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:17.195201 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" event={"ID":"174b1886-1f98-47a8-8463-b7f3539be035","Type":"ContainerDied","Data":"09d0cbc31753a1809d83cae21d32437f8038eb57943ed223144cc54fed50e347"} Nov 01 00:44:18.368526 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.368495 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:18.512008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.511984 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-xks-ca\") pod \"174b1886-1f98-47a8-8463-b7f3539be035\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " Nov 01 00:44:18.512187 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.512026 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-openshift-ca\") pod \"174b1886-1f98-47a8-8463-b7f3539be035\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " Nov 01 00:44:18.512187 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.512068 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174b1886-1f98-47a8-8463-b7f3539be035-kube-api-access-v2bdk\" (UniqueName: \"kubernetes.io/projected/174b1886-1f98-47a8-8463-b7f3539be035-kube-api-access-v2bdk\") pod \"174b1886-1f98-47a8-8463-b7f3539be035\" (UID: \"174b1886-1f98-47a8-8463-b7f3539be035\") " Nov 01 00:44:18.512433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.512413 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-openshift-ca" pod "174b1886-1f98-47a8-8463-b7f3539be035" (UID: "174b1886-1f98-47a8-8463-b7f3539be035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:18.514399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.514365 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174b1886-1f98-47a8-8463-b7f3539be035-kube-api-access-v2bdk" pod "174b1886-1f98-47a8-8463-b7f3539be035" (UID: "174b1886-1f98-47a8-8463-b7f3539be035"). InnerVolumeSpecName "kube-api-access-v2bdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:18.514510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.514430 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-xks-ca" pod "174b1886-1f98-47a8-8463-b7f3539be035" (UID: "174b1886-1f98-47a8-8463-b7f3539be035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:44:18.613217 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.613183 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bdk\" (UniqueName: \"kubernetes.io/projected/174b1886-1f98-47a8-8463-b7f3539be035-kube-api-access-v2bdk\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:18.613217 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.613215 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:18.613394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.613228 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b1886-1f98-47a8-8463-b7f3539be035-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:18.744522 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.744492 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5"] Nov 01 00:44:18.745057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.745042 4910 cpu_manager.go:414] "containerMap: removing container" podUID="174b1886-1f98-47a8-8463-b7f3539be035" containerName="revoke-keys" Nov 01 00:44:18.745118 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.745060 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174b1886-1f98-47a8-8463-b7f3539be035" containerName="revoke-keys" Nov 01 00:44:18.745158 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.745150 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="174b1886-1f98-47a8-8463-b7f3539be035" containerName="revoke-keys" Nov 01 00:44:18.749257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.749236 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.804820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.804730 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-32439f-openshift-default-dockercfg-n9rr6\"" Nov 01 00:44:18.826252 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.826222 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5"] Nov 01 00:44:18.917477 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917441 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.917681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917484 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.917681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917565 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62a8809-b3d0-4475-ac5d-b58536c47600-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.917681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917601 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.917878 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917795 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.917878 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917849 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.917975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917897 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.917975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917927 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:18.918056 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:18.917982 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5jx\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-kube-api-access-4z5jx\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019051 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019091 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019122 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5jx\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-kube-api-access-4z5jx\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019146 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019168 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019201 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62a8809-b3d0-4475-ac5d-b58536c47600-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019222 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019561 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019283 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019561 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019332 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019572 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019676 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019818 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019768 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.019878 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019855 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.020014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.019995 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62a8809-b3d0-4475-ac5d-b58536c47600-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.022151 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.022132 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.022459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.022439 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.080518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.080447 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.080682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.080655 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5jx\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-kube-api-access-4z5jx\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.209789 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.209756 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" Nov 01 00:44:19.209789 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.209778 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74" event={"ID":"174b1886-1f98-47a8-8463-b7f3539be035","Type":"ContainerDied","Data":"368e3bbdbac6b57378889942b1d206e9297f972201db6f53e0e79ca68d5ae0a4"} Nov 01 00:44:19.209985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.209821 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368e3bbdbac6b57378889942b1d206e9297f972201db6f53e0e79ca68d5ae0a4" Nov 01 00:44:19.367377 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.367294 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:19.797440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.797407 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5"] Nov 01 00:44:19.798655 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:19.798602 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62a8809_b3d0_4475_ac5d_b58536c47600.slice/crio-fae3f2f7e9e872df6e4e275f8426ce81c4c6f607da0bbc0689a000fa2ae4cd08 WatchSource:0}: Error finding container fae3f2f7e9e872df6e4e275f8426ce81c4c6f607da0bbc0689a000fa2ae4cd08: Status 404 returned error can't find the container with id fae3f2f7e9e872df6e4e275f8426ce81c4c6f607da0bbc0689a000fa2ae4cd08 Nov 01 00:44:19.801162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.801120 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:19.801285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.801268 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:19.801411 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:19.801393 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:20.218365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:20.218276 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" event={"ID":"a62a8809-b3d0-4475-ac5d-b58536c47600","Type":"ContainerStarted","Data":"eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185"} Nov 01 00:44:20.218365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:20.218330 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" event={"ID":"a62a8809-b3d0-4475-ac5d-b58536c47600","Type":"ContainerStarted","Data":"fae3f2f7e9e872df6e4e275f8426ce81c4c6f607da0bbc0689a000fa2ae4cd08"} Nov 01 00:44:20.341271 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:20.341221 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podStartSLOduration=2.341209208 podStartE2EDuration="2.341209208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:44:20.338002835 +0000 UTC m=+1250.317585170" watchObservedRunningTime="2026-11-01 00:44:20.341209208 +0000 UTC m=+1250.320791542" Nov 01 00:44:20.367959 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:20.367926 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:20.369663 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:20.369613 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:44:20.369787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:20.369700 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 00:44:21.371670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.369090 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:44:21.371670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.369171 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 00:44:21.475109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.475067 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h"] Nov 01 00:44:21.482177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.482149 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.527864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.527839 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h"] Nov 01 00:44:21.545547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.545518 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7nfh\" (UniqueName: \"kubernetes.io/projected/101d3974-26c6-49f4-9884-374f485be157-kube-api-access-t7nfh\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.545704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.545558 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101d3974-26c6-49f4-9884-374f485be157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.545881 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.545858 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101d3974-26c6-49f4-9884-374f485be157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.575643 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.573843 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:44:21.590856 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.590827 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:44:21.647433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.647348 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101d3974-26c6-49f4-9884-374f485be157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.647433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.647386 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7nfh\" (UniqueName: \"kubernetes.io/projected/101d3974-26c6-49f4-9884-374f485be157-kube-api-access-t7nfh\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.647433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.647408 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101d3974-26c6-49f4-9884-374f485be157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.648053 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.648024 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101d3974-26c6-49f4-9884-374f485be157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.655646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.650052 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101d3974-26c6-49f4-9884-374f485be157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.714646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.709395 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7nfh\" (UniqueName: \"kubernetes.io/projected/101d3974-26c6-49f4-9884-374f485be157-kube-api-access-t7nfh\") pod \"maas-api-revoke-sub-fde5bcb8-tjj2h\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:21.797152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:21.797123 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:22.215817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:22.215782 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h"] Nov 01 00:44:22.221156 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:22.221117 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101d3974_26c6_49f4_9884_374f485be157.slice/crio-a904667ca7c77a375fbb4b9946d46f2efa4a9ea40d034eb1b7a230141bcbb9d6 WatchSource:0}: Error finding container a904667ca7c77a375fbb4b9946d46f2efa4a9ea40d034eb1b7a230141bcbb9d6: Status 404 returned error can't find the container with id a904667ca7c77a375fbb4b9946d46f2efa4a9ea40d034eb1b7a230141bcbb9d6 Nov 01 00:44:22.235686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:22.235656 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" event={"ID":"101d3974-26c6-49f4-9884-374f485be157","Type":"ContainerStarted","Data":"a904667ca7c77a375fbb4b9946d46f2efa4a9ea40d034eb1b7a230141bcbb9d6"} Nov 01 00:44:22.371047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:22.371011 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:44:22.371306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:22.371273 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 00:44:23.245970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:23.245897 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" event={"ID":"101d3974-26c6-49f4-9884-374f485be157","Type":"ContainerStarted","Data":"3698bde14f2bc731c7817921aa5bace28ac4bf2935d9ac5756d2e4411653b316"} Nov 01 00:44:23.373648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:23.372498 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" podStartSLOduration=2.372478373 podStartE2EDuration="2.372478373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:44:23.370017505 +0000 UTC m=+1253.349599842" watchObservedRunningTime="2026-11-01 00:44:23.372478373 +0000 UTC m=+1253.352060709" Nov 01 00:44:23.374521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:23.374136 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:44:23.374521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:23.374184 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 00:44:24.277123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:24.277085 4910 generic.go:356] "Generic (PLEG): container finished" podID="101d3974-26c6-49f4-9884-374f485be157" containerID="3698bde14f2bc731c7817921aa5bace28ac4bf2935d9ac5756d2e4411653b316" exitCode=0 Nov 01 00:44:24.277528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:24.277318 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" event={"ID":"101d3974-26c6-49f4-9884-374f485be157","Type":"ContainerDied","Data":"3698bde14f2bc731c7817921aa5bace28ac4bf2935d9ac5756d2e4411653b316"} Nov 01 00:44:24.375653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:24.370969 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:44:24.375653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:24.371026 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 00:44:25.375653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.372991 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 01 00:44:25.375653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.373054 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 01 00:44:25.443026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.443003 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:25.506843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.506814 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101d3974-26c6-49f4-9884-374f485be157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101d3974-26c6-49f4-9884-374f485be157-maas-api-openshift-ca\") pod \"101d3974-26c6-49f4-9884-374f485be157\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " Nov 01 00:44:25.506971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.506893 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101d3974-26c6-49f4-9884-374f485be157-kube-api-access-t7nfh\" (UniqueName: \"kubernetes.io/projected/101d3974-26c6-49f4-9884-374f485be157-kube-api-access-t7nfh\") pod \"101d3974-26c6-49f4-9884-374f485be157\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " Nov 01 00:44:25.507022 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.506973 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101d3974-26c6-49f4-9884-374f485be157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101d3974-26c6-49f4-9884-374f485be157-maas-api-xks-ca\") pod \"101d3974-26c6-49f4-9884-374f485be157\" (UID: \"101d3974-26c6-49f4-9884-374f485be157\") " Nov 01 00:44:25.507185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.507164 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101d3974-26c6-49f4-9884-374f485be157-maas-api-openshift-ca" pod "101d3974-26c6-49f4-9884-374f485be157" (UID: "101d3974-26c6-49f4-9884-374f485be157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:25.509191 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.509171 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101d3974-26c6-49f4-9884-374f485be157-maas-api-xks-ca" pod "101d3974-26c6-49f4-9884-374f485be157" (UID: "101d3974-26c6-49f4-9884-374f485be157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:44:25.509191 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.509184 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101d3974-26c6-49f4-9884-374f485be157-kube-api-access-t7nfh" pod "101d3974-26c6-49f4-9884-374f485be157" (UID: "101d3974-26c6-49f4-9884-374f485be157"). InnerVolumeSpecName "kube-api-access-t7nfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:25.608282 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.608212 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101d3974-26c6-49f4-9884-374f485be157-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:25.608282 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.608236 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101d3974-26c6-49f4-9884-374f485be157-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:25.608282 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:25.608248 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7nfh\" (UniqueName: \"kubernetes.io/projected/101d3974-26c6-49f4-9884-374f485be157-kube-api-access-t7nfh\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:26.292189 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:26.292156 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" Nov 01 00:44:26.292344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:26.292182 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h" event={"ID":"101d3974-26c6-49f4-9884-374f485be157","Type":"ContainerDied","Data":"a904667ca7c77a375fbb4b9946d46f2efa4a9ea40d034eb1b7a230141bcbb9d6"} Nov 01 00:44:26.292344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:26.292236 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a904667ca7c77a375fbb4b9946d46f2efa4a9ea40d034eb1b7a230141bcbb9d6" Nov 01 00:44:27.303184 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:27.303144 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:27.304374 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:27.304337 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:27.305639 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:27.305601 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 00:44:30.680589 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.680513 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n"] Nov 01 00:44:30.681011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.680997 4910 cpu_manager.go:414] "containerMap: removing container" podUID="101d3974-26c6-49f4-9884-374f485be157" containerName="revoke-keys" Nov 01 00:44:30.681073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.681013 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101d3974-26c6-49f4-9884-374f485be157" containerName="revoke-keys" Nov 01 00:44:30.681119 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.681109 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="101d3974-26c6-49f4-9884-374f485be157" containerName="revoke-keys" Nov 01 00:44:30.685502 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.685483 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.741961 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.741935 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n"] Nov 01 00:44:30.757165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.757133 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.757335 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.757317 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.757400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.757357 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spg4q\" (UniqueName: \"kubernetes.io/projected/1eb7225d-ef79-4657-85a5-674f4acfeb87-kube-api-access-spg4q\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.788323 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.788299 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:44:30.788447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.788431 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:44:30.858414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.858375 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.858414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.858414 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spg4q\" (UniqueName: \"kubernetes.io/projected/1eb7225d-ef79-4657-85a5-674f4acfeb87-kube-api-access-spg4q\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.858686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.858489 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.859291 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.859268 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.861041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.861019 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.919470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.919445 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spg4q\" (UniqueName: \"kubernetes.io/projected/1eb7225d-ef79-4657-85a5-674f4acfeb87-kube-api-access-spg4q\") pod \"maas-api-revoke-sub-f732c126-js62n\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:30.998783 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:30.998764 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:31.378640 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:31.378598 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n"] Nov 01 00:44:31.379052 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:31.379021 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb7225d_ef79_4657_85a5_674f4acfeb87.slice/crio-db98185bddde964cad1ecdf87253a938f1ccfb98dd89ab7358e33a5831d2f80b WatchSource:0}: Error finding container db98185bddde964cad1ecdf87253a938f1ccfb98dd89ab7358e33a5831d2f80b: Status 404 returned error can't find the container with id db98185bddde964cad1ecdf87253a938f1ccfb98dd89ab7358e33a5831d2f80b Nov 01 00:44:32.338818 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:32.338781 4910 generic.go:356] "Generic (PLEG): container finished" podID="1eb7225d-ef79-4657-85a5-674f4acfeb87" containerID="b3aaee0379bdde739a58cb278223e08b2c2e9293515fc1e0e703abe0a32d9d0f" exitCode=0 Nov 01 00:44:32.339247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:32.338828 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" event={"ID":"1eb7225d-ef79-4657-85a5-674f4acfeb87","Type":"ContainerDied","Data":"b3aaee0379bdde739a58cb278223e08b2c2e9293515fc1e0e703abe0a32d9d0f"} Nov 01 00:44:32.339247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:32.338892 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" event={"ID":"1eb7225d-ef79-4657-85a5-674f4acfeb87","Type":"ContainerStarted","Data":"db98185bddde964cad1ecdf87253a938f1ccfb98dd89ab7358e33a5831d2f80b"} Nov 01 00:44:33.277026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.276992 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf"] Nov 01 00:44:33.281972 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.281950 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.334475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.334448 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-22d7b2-openshift-default-dockercfg-lhnwj\"" Nov 01 00:44:33.379602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.379563 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.379647 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.379771 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.379853 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.379883 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.380006 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxx69\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-kube-api-access-rxx69\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380256 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.380058 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380256 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.380078 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.380256 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.380120 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.390033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.389996 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf"] Nov 01 00:44:33.480733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.480702 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.480897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.480768 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxx69\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-kube-api-access-rxx69\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.480897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.480805 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.480897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.480836 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481067 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.480900 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481067 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.480939 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481067 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.480971 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481067 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.481032 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.481107 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.481124 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.481176 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.481351 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.481418 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.481571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.481545 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.483497 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.483480 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.483652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.483611 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.512130 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.512109 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:33.542434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.542371 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.542611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.542586 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxx69\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-kube-api-access-rxx69\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.581763 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.581725 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb7225d-ef79-4657-85a5-674f4acfeb87-kube-api-access-spg4q\" (UniqueName: \"kubernetes.io/projected/1eb7225d-ef79-4657-85a5-674f4acfeb87-kube-api-access-spg4q\") pod \"1eb7225d-ef79-4657-85a5-674f4acfeb87\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " Nov 01 00:44:33.581918 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.581865 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-openshift-ca\") pod \"1eb7225d-ef79-4657-85a5-674f4acfeb87\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " Nov 01 00:44:33.581969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.581915 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-xks-ca\") pod \"1eb7225d-ef79-4657-85a5-674f4acfeb87\" (UID: \"1eb7225d-ef79-4657-85a5-674f4acfeb87\") " Nov 01 00:44:33.582233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.582205 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-openshift-ca" pod "1eb7225d-ef79-4657-85a5-674f4acfeb87" (UID: "1eb7225d-ef79-4657-85a5-674f4acfeb87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:33.584120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.584093 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-xks-ca" pod "1eb7225d-ef79-4657-85a5-674f4acfeb87" (UID: "1eb7225d-ef79-4657-85a5-674f4acfeb87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:44:33.584120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.584097 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb7225d-ef79-4657-85a5-674f4acfeb87-kube-api-access-spg4q" pod "1eb7225d-ef79-4657-85a5-674f4acfeb87" (UID: "1eb7225d-ef79-4657-85a5-674f4acfeb87"). InnerVolumeSpecName "kube-api-access-spg4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:33.595325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.595306 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:33.683529 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.683490 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:33.683529 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.683531 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb7225d-ef79-4657-85a5-674f4acfeb87-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:33.683736 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:33.683547 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spg4q\" (UniqueName: \"kubernetes.io/projected/1eb7225d-ef79-4657-85a5-674f4acfeb87-kube-api-access-spg4q\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:34.031244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.031215 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf"] Nov 01 00:44:34.031429 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:34.031408 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8794b0b6_eadf_4bd6_ade4_eff3517b6552.slice/crio-66f4a49857227ee6a8993628c8bd60820065c6be5d57aa7af1b3d2da45a732e7 WatchSource:0}: Error finding container 66f4a49857227ee6a8993628c8bd60820065c6be5d57aa7af1b3d2da45a732e7: Status 404 returned error can't find the container with id 66f4a49857227ee6a8993628c8bd60820065c6be5d57aa7af1b3d2da45a732e7 Nov 01 00:44:34.033820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.033785 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:34.033934 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.033869 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:34.033934 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.033917 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:34.353085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.352996 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" Nov 01 00:44:34.353085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.353029 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n" event={"ID":"1eb7225d-ef79-4657-85a5-674f4acfeb87","Type":"ContainerDied","Data":"db98185bddde964cad1ecdf87253a938f1ccfb98dd89ab7358e33a5831d2f80b"} Nov 01 00:44:34.353315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.353089 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db98185bddde964cad1ecdf87253a938f1ccfb98dd89ab7358e33a5831d2f80b" Nov 01 00:44:34.355212 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.355183 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" event={"ID":"8794b0b6-eadf-4bd6-ade4-eff3517b6552","Type":"ContainerStarted","Data":"17e056297ca12db1bee566bfef3fd31884b750750dc6c759e53f6d5728979d34"} Nov 01 00:44:34.355365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.355236 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" event={"ID":"8794b0b6-eadf-4bd6-ade4-eff3517b6552","Type":"ContainerStarted","Data":"66f4a49857227ee6a8993628c8bd60820065c6be5d57aa7af1b3d2da45a732e7"} Nov 01 00:44:34.475568 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.475512 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podStartSLOduration=1.4754969980000001 podStartE2EDuration="1.475496998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:44:34.475378168 +0000 UTC m=+1264.454960507" watchObservedRunningTime="2026-11-01 00:44:34.475496998 +0000 UTC m=+1264.455079356" Nov 01 00:44:34.595787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:34.595755 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:35.597210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:35.597177 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:44:35.597692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:35.597233 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 00:44:36.596550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:36.596511 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:44:36.596872 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:36.596562 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 00:44:37.596211 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:37.596173 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:44:37.596602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:37.596256 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 00:44:38.595861 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:38.595830 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:44:38.596038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:38.595888 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 00:44:39.595986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.595953 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 00:44:39.596373 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.596011 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 00:44:39.640306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.640274 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs"] Nov 01 00:44:39.640830 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.640817 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1eb7225d-ef79-4657-85a5-674f4acfeb87" containerName="revoke-keys" Nov 01 00:44:39.640877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.640831 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb7225d-ef79-4657-85a5-674f4acfeb87" containerName="revoke-keys" Nov 01 00:44:39.640932 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.640922 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb7225d-ef79-4657-85a5-674f4acfeb87" containerName="revoke-keys" Nov 01 00:44:39.648230 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.648195 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.704153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.704124 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs"] Nov 01 00:44:39.705574 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.705548 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:44:39.745071 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.745044 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.745071 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.745075 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6x76\" (UniqueName: \"kubernetes.io/projected/db5e9fc1-f214-415a-8eeb-fc1954933214-kube-api-access-r6x76\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.745268 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.745118 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.775950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.774845 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:44:39.845967 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.845938 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.846176 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.846064 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.846176 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.846090 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6x76\" (UniqueName: \"kubernetes.io/projected/db5e9fc1-f214-415a-8eeb-fc1954933214-kube-api-access-r6x76\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.846560 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.846540 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.848335 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.848314 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.907216 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.907191 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6x76\" (UniqueName: \"kubernetes.io/projected/db5e9fc1-f214-415a-8eeb-fc1954933214-kube-api-access-r6x76\") pod \"maas-api-revoke-sub-036b33c5-dlgxs\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:39.983576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:39.983538 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:40.374390 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:40.374361 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs"] Nov 01 00:44:40.376175 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:40.376134 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5e9fc1_f214_415a_8eeb_fc1954933214.slice/crio-d7ab8b40cf298bc5c0ec74c2350575e6c4656e03a0c4d3dfe4d44c693f9f6102 WatchSource:0}: Error finding container d7ab8b40cf298bc5c0ec74c2350575e6c4656e03a0c4d3dfe4d44c693f9f6102: Status 404 returned error can't find the container with id d7ab8b40cf298bc5c0ec74c2350575e6c4656e03a0c4d3dfe4d44c693f9f6102 Nov 01 00:44:40.413947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:40.413912 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" event={"ID":"db5e9fc1-f214-415a-8eeb-fc1954933214","Type":"ContainerStarted","Data":"d7ab8b40cf298bc5c0ec74c2350575e6c4656e03a0c4d3dfe4d44c693f9f6102"} Nov 01 00:44:40.744617 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:40.744584 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:40.745656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:40.745213 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:40.746464 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:40.746446 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 00:44:41.421036 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.421000 4910 generic.go:356] "Generic (PLEG): container finished" podID="db5e9fc1-f214-415a-8eeb-fc1954933214" containerID="36d3ee2c45a29595e1fa2b11292b14fa12b197a85a7a7a85c244ec7e98ad6e31" exitCode=0 Nov 01 00:44:41.421168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.421089 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" event={"ID":"db5e9fc1-f214-415a-8eeb-fc1954933214","Type":"ContainerDied","Data":"36d3ee2c45a29595e1fa2b11292b14fa12b197a85a7a7a85c244ec7e98ad6e31"} Nov 01 00:44:41.568144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.568059 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn"] Nov 01 00:44:41.572086 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.572062 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.651981 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.651953 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn"] Nov 01 00:44:41.664007 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.663983 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.664117 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.664105 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbl4\" (UniqueName: \"kubernetes.io/projected/8d40c94a-022e-4346-8b9a-d6eb735daa2a-kube-api-access-wvbl4\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.664179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.664130 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.764721 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.764689 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.765179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.764762 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbl4\" (UniqueName: \"kubernetes.io/projected/8d40c94a-022e-4346-8b9a-d6eb735daa2a-kube-api-access-wvbl4\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.765179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.764783 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.765424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.765406 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.767242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.767217 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.825851 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.825791 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbl4\" (UniqueName: \"kubernetes.io/projected/8d40c94a-022e-4346-8b9a-d6eb735daa2a-kube-api-access-wvbl4\") pod \"maas-api-revoke-sub-fcf8a334-8s6kn\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:41.885704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:41.885680 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:42.268475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.268443 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn"] Nov 01 00:44:42.272043 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:42.272015 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d40c94a_022e_4346_8b9a_d6eb735daa2a.slice/crio-67f2a2d8389ea93b8be98f9f59fdc664d6ffede3b566fa8441a98d9b1527786c WatchSource:0}: Error finding container 67f2a2d8389ea93b8be98f9f59fdc664d6ffede3b566fa8441a98d9b1527786c: Status 404 returned error can't find the container with id 67f2a2d8389ea93b8be98f9f59fdc664d6ffede3b566fa8441a98d9b1527786c Nov 01 00:44:42.429153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.429102 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" event={"ID":"8d40c94a-022e-4346-8b9a-d6eb735daa2a","Type":"ContainerStarted","Data":"67f2a2d8389ea93b8be98f9f59fdc664d6ffede3b566fa8441a98d9b1527786c"} Nov 01 00:44:42.567311 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.567276 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:42.674857 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.674831 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-xks-ca\") pod \"db5e9fc1-f214-415a-8eeb-fc1954933214\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " Nov 01 00:44:42.674989 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.674951 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-openshift-ca\") pod \"db5e9fc1-f214-415a-8eeb-fc1954933214\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " Nov 01 00:44:42.675082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.675064 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5e9fc1-f214-415a-8eeb-fc1954933214-kube-api-access-r6x76\" (UniqueName: \"kubernetes.io/projected/db5e9fc1-f214-415a-8eeb-fc1954933214-kube-api-access-r6x76\") pod \"db5e9fc1-f214-415a-8eeb-fc1954933214\" (UID: \"db5e9fc1-f214-415a-8eeb-fc1954933214\") " Nov 01 00:44:42.675328 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.675302 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-openshift-ca" pod "db5e9fc1-f214-415a-8eeb-fc1954933214" (UID: "db5e9fc1-f214-415a-8eeb-fc1954933214"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:42.675442 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.675418 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:42.676968 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.676946 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-xks-ca" pod "db5e9fc1-f214-415a-8eeb-fc1954933214" (UID: "db5e9fc1-f214-415a-8eeb-fc1954933214"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:44:42.677231 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.677206 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5e9fc1-f214-415a-8eeb-fc1954933214-kube-api-access-r6x76" pod "db5e9fc1-f214-415a-8eeb-fc1954933214" (UID: "db5e9fc1-f214-415a-8eeb-fc1954933214"). InnerVolumeSpecName "kube-api-access-r6x76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:42.777021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.776980 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e9fc1-f214-415a-8eeb-fc1954933214-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:42.777021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:42.777014 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6x76\" (UniqueName: \"kubernetes.io/projected/db5e9fc1-f214-415a-8eeb-fc1954933214-kube-api-access-r6x76\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:43.103015 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:44:43.102982 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:44:43.438058 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:43.438027 4910 generic.go:356] "Generic (PLEG): container finished" podID="8d40c94a-022e-4346-8b9a-d6eb735daa2a" containerID="041dd3ca133e0d80358322334ade4d907749cd0dc5e14366b68174e45c62b398" exitCode=0 Nov 01 00:44:43.438238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:43.438105 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" event={"ID":"8d40c94a-022e-4346-8b9a-d6eb735daa2a","Type":"ContainerDied","Data":"041dd3ca133e0d80358322334ade4d907749cd0dc5e14366b68174e45c62b398"} Nov 01 00:44:43.439687 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:43.439665 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" event={"ID":"db5e9fc1-f214-415a-8eeb-fc1954933214","Type":"ContainerDied","Data":"d7ab8b40cf298bc5c0ec74c2350575e6c4656e03a0c4d3dfe4d44c693f9f6102"} Nov 01 00:44:43.439822 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:43.439691 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs" Nov 01 00:44:43.439822 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:43.439699 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ab8b40cf298bc5c0ec74c2350575e6c4656e03a0c4d3dfe4d44c693f9f6102" Nov 01 00:44:44.613769 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.613736 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:44.695749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.695718 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-xks-ca\") pod \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " Nov 01 00:44:44.695959 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.695797 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-openshift-ca\") pod \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " Nov 01 00:44:44.695959 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.695865 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d40c94a-022e-4346-8b9a-d6eb735daa2a-kube-api-access-wvbl4\" (UniqueName: \"kubernetes.io/projected/8d40c94a-022e-4346-8b9a-d6eb735daa2a-kube-api-access-wvbl4\") pod \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\" (UID: \"8d40c94a-022e-4346-8b9a-d6eb735daa2a\") " Nov 01 00:44:44.696204 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.696177 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-openshift-ca" pod "8d40c94a-022e-4346-8b9a-d6eb735daa2a" (UID: "8d40c94a-022e-4346-8b9a-d6eb735daa2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:44.696310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.696292 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:44.698169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.698108 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-xks-ca" pod "8d40c94a-022e-4346-8b9a-d6eb735daa2a" (UID: "8d40c94a-022e-4346-8b9a-d6eb735daa2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:44:44.698441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.698419 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d40c94a-022e-4346-8b9a-d6eb735daa2a-kube-api-access-wvbl4" pod "8d40c94a-022e-4346-8b9a-d6eb735daa2a" (UID: "8d40c94a-022e-4346-8b9a-d6eb735daa2a"). InnerVolumeSpecName "kube-api-access-wvbl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:44.797379 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.797352 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d40c94a-022e-4346-8b9a-d6eb735daa2a-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:44.797379 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:44.797380 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvbl4\" (UniqueName: \"kubernetes.io/projected/8d40c94a-022e-4346-8b9a-d6eb735daa2a-kube-api-access-wvbl4\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:45.453432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:45.453390 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" Nov 01 00:44:45.453432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:45.453414 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn" event={"ID":"8d40c94a-022e-4346-8b9a-d6eb735daa2a","Type":"ContainerDied","Data":"67f2a2d8389ea93b8be98f9f59fdc664d6ffede3b566fa8441a98d9b1527786c"} Nov 01 00:44:45.453684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:45.453463 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f2a2d8389ea93b8be98f9f59fdc664d6ffede3b566fa8441a98d9b1527786c" Nov 01 00:44:49.614207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.614174 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk"] Nov 01 00:44:49.614670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.614657 4910 cpu_manager.go:414] "containerMap: removing container" podUID="db5e9fc1-f214-415a-8eeb-fc1954933214" containerName="revoke-keys" Nov 01 00:44:49.614731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.614671 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5e9fc1-f214-415a-8eeb-fc1954933214" containerName="revoke-keys" Nov 01 00:44:49.614731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.614684 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8d40c94a-022e-4346-8b9a-d6eb735daa2a" containerName="revoke-keys" Nov 01 00:44:49.614731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.614689 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d40c94a-022e-4346-8b9a-d6eb735daa2a" containerName="revoke-keys" Nov 01 00:44:49.614834 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.614789 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5e9fc1-f214-415a-8eeb-fc1954933214" containerName="revoke-keys" Nov 01 00:44:49.614834 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.614798 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d40c94a-022e-4346-8b9a-d6eb735daa2a" containerName="revoke-keys" Nov 01 00:44:49.617993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.617979 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.670712 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.670671 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4143b7-openshift-default-dockercfg-fhj7l\"" Nov 01 00:44:49.725129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.725101 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk"] Nov 01 00:44:49.754986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.754957 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755000 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a66a46-af99-4275-9ed6-1c513dd1f327-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755137 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755192 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755226 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755320 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755301 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrx6\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-kube-api-access-jfrx6\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755352 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755432 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.755539 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.755524 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856332 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856304 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856347 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrx6\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-kube-api-access-jfrx6\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856387 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856412 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856716 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856571 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856772 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856716 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856772 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856741 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856772 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856747 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a66a46-af99-4275-9ed6-1c513dd1f327-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856802 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856840 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.856917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.856896 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.857156 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.857132 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.857265 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.857160 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.857433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.857412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a66a46-af99-4275-9ed6-1c513dd1f327-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.859746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.859721 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.859883 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.859865 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.917771 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.917692 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.917903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.917805 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrx6\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-kube-api-access-jfrx6\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:49.931595 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:49.931572 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:50.361178 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.361153 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk"] Nov 01 00:44:50.361848 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:50.361820 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a66a46_af99_4275_9ed6_1c513dd1f327.slice/crio-3f2ec702dcd9ca01ff878108f4b80012808f4ba1355945d6e65e1e91bd73397b WatchSource:0}: Error finding container 3f2ec702dcd9ca01ff878108f4b80012808f4ba1355945d6e65e1e91bd73397b: Status 404 returned error can't find the container with id 3f2ec702dcd9ca01ff878108f4b80012808f4ba1355945d6e65e1e91bd73397b Nov 01 00:44:50.364091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.364058 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:50.364194 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.364128 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:50.364194 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.364155 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:44:50.491641 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.491591 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" event={"ID":"e0a66a46-af99-4275-9ed6-1c513dd1f327","Type":"ContainerStarted","Data":"938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d"} Nov 01 00:44:50.491819 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.491683 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" event={"ID":"e0a66a46-af99-4275-9ed6-1c513dd1f327","Type":"ContainerStarted","Data":"3f2ec702dcd9ca01ff878108f4b80012808f4ba1355945d6e65e1e91bd73397b"} Nov 01 00:44:50.615720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.615589 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podStartSLOduration=1.6155691989999998 podStartE2EDuration="1.615569199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:44:50.612124754 +0000 UTC m=+1280.591707088" watchObservedRunningTime="2026-11-01 00:44:50.615569199 +0000 UTC m=+1280.595151534" Nov 01 00:44:50.935784 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:50.935701 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:51.933141 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:51.933105 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:44:51.933533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:51.933167 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 00:44:52.380692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.380640 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v"] Nov 01 00:44:52.385528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.385503 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.432836 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.432811 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v"] Nov 01 00:44:52.478678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.478642 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:44:52.478814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.478717 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:44:52.492169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.492142 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.492283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.492190 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.492327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.492315 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pb8h\" (UniqueName: \"kubernetes.io/projected/904f3f42-3c45-4490-b25d-996ba6b26bbf-kube-api-access-2pb8h\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.593091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.593058 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pb8h\" (UniqueName: \"kubernetes.io/projected/904f3f42-3c45-4490-b25d-996ba6b26bbf-kube-api-access-2pb8h\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.593258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.593170 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.593258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.593237 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.594172 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.594145 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.595826 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.595805 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.680710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.679661 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pb8h\" (UniqueName: \"kubernetes.io/projected/904f3f42-3c45-4490-b25d-996ba6b26bbf-kube-api-access-2pb8h\") pod \"maas-api-revoke-sub-31fbebc0-t5q9v\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.700514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.700481 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:52.932725 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.932648 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:44:52.932725 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:52.932709 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 00:44:53.106403 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:53.106374 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v"] Nov 01 00:44:53.107162 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:44:53.107138 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904f3f42_3c45_4490_b25d_996ba6b26bbf.slice/crio-20a03add853e9b6b9284ac0a6c988170324b02e17443254b31fe66750a59b0b1 WatchSource:0}: Error finding container 20a03add853e9b6b9284ac0a6c988170324b02e17443254b31fe66750a59b0b1: Status 404 returned error can't find the container with id 20a03add853e9b6b9284ac0a6c988170324b02e17443254b31fe66750a59b0b1 Nov 01 00:44:53.516159 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:53.516110 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" event={"ID":"904f3f42-3c45-4490-b25d-996ba6b26bbf","Type":"ContainerStarted","Data":"a7a0763f72c124d166f1881a36816efef984829a38b20ce3c38f7cd944309ea1"} Nov 01 00:44:53.516325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:53.516183 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" event={"ID":"904f3f42-3c45-4490-b25d-996ba6b26bbf","Type":"ContainerStarted","Data":"20a03add853e9b6b9284ac0a6c988170324b02e17443254b31fe66750a59b0b1"} Nov 01 00:44:53.637907 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:53.637798 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" podStartSLOduration=1.637779299 podStartE2EDuration="1.637779299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:44:53.637357029 +0000 UTC m=+1283.616939365" watchObservedRunningTime="2026-11-01 00:44:53.637779299 +0000 UTC m=+1283.617361635" Nov 01 00:44:53.932869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:53.932788 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:44:53.932869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:53.932840 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 00:44:54.524377 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:54.524348 4910 generic.go:356] "Generic (PLEG): container finished" podID="904f3f42-3c45-4490-b25d-996ba6b26bbf" containerID="a7a0763f72c124d166f1881a36816efef984829a38b20ce3c38f7cd944309ea1" exitCode=0 Nov 01 00:44:54.524812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:54.524435 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" event={"ID":"904f3f42-3c45-4490-b25d-996ba6b26bbf","Type":"ContainerDied","Data":"a7a0763f72c124d166f1881a36816efef984829a38b20ce3c38f7cd944309ea1"} Nov 01 00:44:54.932749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:54.932667 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:44:54.932749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:54.932714 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 00:44:55.721341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.721318 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:55.833024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.832943 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-xks-ca\") pod \"904f3f42-3c45-4490-b25d-996ba6b26bbf\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " Nov 01 00:44:55.833177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.833075 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904f3f42-3c45-4490-b25d-996ba6b26bbf-kube-api-access-2pb8h\" (UniqueName: \"kubernetes.io/projected/904f3f42-3c45-4490-b25d-996ba6b26bbf-kube-api-access-2pb8h\") pod \"904f3f42-3c45-4490-b25d-996ba6b26bbf\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " Nov 01 00:44:55.833177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.833100 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-openshift-ca\") pod \"904f3f42-3c45-4490-b25d-996ba6b26bbf\" (UID: \"904f3f42-3c45-4490-b25d-996ba6b26bbf\") " Nov 01 00:44:55.833523 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.833500 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-openshift-ca" pod "904f3f42-3c45-4490-b25d-996ba6b26bbf" (UID: "904f3f42-3c45-4490-b25d-996ba6b26bbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:44:55.835168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.835140 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-xks-ca" pod "904f3f42-3c45-4490-b25d-996ba6b26bbf" (UID: "904f3f42-3c45-4490-b25d-996ba6b26bbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:44:55.835340 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.835322 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904f3f42-3c45-4490-b25d-996ba6b26bbf-kube-api-access-2pb8h" pod "904f3f42-3c45-4490-b25d-996ba6b26bbf" (UID: "904f3f42-3c45-4490-b25d-996ba6b26bbf"). InnerVolumeSpecName "kube-api-access-2pb8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:44:55.932205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.932120 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 00:44:55.932359 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.932197 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 00:44:55.934817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.934786 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pb8h\" (UniqueName: \"kubernetes.io/projected/904f3f42-3c45-4490-b25d-996ba6b26bbf-kube-api-access-2pb8h\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:55.934817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.934813 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:55.934990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:55.934824 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904f3f42-3c45-4490-b25d-996ba6b26bbf-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:44:56.540441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:56.540410 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" Nov 01 00:44:56.540441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:56.540433 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v" event={"ID":"904f3f42-3c45-4490-b25d-996ba6b26bbf","Type":"ContainerDied","Data":"20a03add853e9b6b9284ac0a6c988170324b02e17443254b31fe66750a59b0b1"} Nov 01 00:44:56.540747 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:56.540489 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a03add853e9b6b9284ac0a6c988170324b02e17443254b31fe66750a59b0b1" Nov 01 00:44:56.969454 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:56.969354 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:56.969909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:56.969890 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:44:56.970588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:44:56.970569 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 00:45:00.177580 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.177545 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7"] Nov 01 00:45:00.178372 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.178352 4910 cpu_manager.go:414] "containerMap: removing container" podUID="904f3f42-3c45-4490-b25d-996ba6b26bbf" containerName="revoke-keys" Nov 01 00:45:00.178452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.178375 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904f3f42-3c45-4490-b25d-996ba6b26bbf" containerName="revoke-keys" Nov 01 00:45:00.178541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.178529 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="904f3f42-3c45-4490-b25d-996ba6b26bbf" containerName="revoke-keys" Nov 01 00:45:00.182568 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.182545 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.182736 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.182548 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c"] Nov 01 00:45:00.187437 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.187411 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh"] Nov 01 00:45:00.187558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.187545 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.195107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.195082 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.226008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.225977 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q"] Nov 01 00:45:00.231003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.230974 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc"] Nov 01 00:45:00.232446 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.232425 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.235123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.235098 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:45:00.237428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.237402 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.280962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.280724 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda98f1c-115c-4625-8f42-8850439f3008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.280962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.280771 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjhv\" (UniqueName: \"kubernetes.io/projected/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-kube-api-access-svjhv\") pod \"maas-api-key-cleanup-29891565-nr76c\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.280962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.280825 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891565-nr76c\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.280962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.280883 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cdlf\" (UniqueName: \"kubernetes.io/projected/fda98f1c-115c-4625-8f42-8850439f3008-kube-api-access-7cdlf\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.280962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.280918 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee4f1a2-9040-4505-8001-5f5a23a2d122-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.281374 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.281001 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mn9\" (UniqueName: \"kubernetes.io/projected/6ee4f1a2-9040-4505-8001-5f5a23a2d122-kube-api-access-22mn9\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.281374 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.281201 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7"] Nov 01 00:45:00.294327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.287090 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c"] Nov 01 00:45:00.294327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.290965 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh"] Nov 01 00:45:00.297678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.296683 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q"] Nov 01 00:45:00.353262 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.353133 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc"] Nov 01 00:45:00.381775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.381728 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mn9\" (UniqueName: \"kubernetes.io/projected/6ee4f1a2-9040-4505-8001-5f5a23a2d122-kube-api-access-22mn9\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.381949 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.381809 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwf8c\" (UniqueName: \"kubernetes.io/projected/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-kube-api-access-qwf8c\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.381949 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.381864 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.382075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.381964 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda98f1c-115c-4625-8f42-8850439f3008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.382075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.382001 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjhv\" (UniqueName: \"kubernetes.io/projected/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-kube-api-access-svjhv\") pod \"maas-api-key-cleanup-29891565-nr76c\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.382075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.382053 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvpj\" (UniqueName: \"kubernetes.io/projected/15066416-112b-4dfd-8761-2cf5116b8021-kube-api-access-9rvpj\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.382206 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.382083 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15066416-112b-4dfd-8761-2cf5116b8021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.382206 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.382112 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891565-nr76c\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.382206 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.382170 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cdlf\" (UniqueName: \"kubernetes.io/projected/fda98f1c-115c-4625-8f42-8850439f3008-kube-api-access-7cdlf\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.382354 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.382208 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee4f1a2-9040-4505-8001-5f5a23a2d122-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.382782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.382758 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda98f1c-115c-4625-8f42-8850439f3008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.383095 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.383073 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee4f1a2-9040-4505-8001-5f5a23a2d122-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.383156 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.383092 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891565-nr76c\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.443261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.443193 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cdlf\" (UniqueName: \"kubernetes.io/projected/fda98f1c-115c-4625-8f42-8850439f3008-kube-api-access-7cdlf\") pod \"maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.443261 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.443234 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mn9\" (UniqueName: \"kubernetes.io/projected/6ee4f1a2-9040-4505-8001-5f5a23a2d122-kube-api-access-22mn9\") pod \"maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.443468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.443440 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjhv\" (UniqueName: \"kubernetes.io/projected/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-kube-api-access-svjhv\") pod \"maas-api-key-cleanup-29891565-nr76c\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.483210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.483176 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.483376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.483247 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvpj\" (UniqueName: \"kubernetes.io/projected/15066416-112b-4dfd-8761-2cf5116b8021-kube-api-access-9rvpj\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.483376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.483275 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15066416-112b-4dfd-8761-2cf5116b8021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.483487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.483422 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwf8c\" (UniqueName: \"kubernetes.io/projected/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-kube-api-access-qwf8c\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.484007 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.483984 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15066416-112b-4dfd-8761-2cf5116b8021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.484114 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.483993 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.501418 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.501392 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:00.512915 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.512889 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:00.520942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.520917 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:00.546346 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.546318 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwf8c\" (UniqueName: \"kubernetes.io/projected/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-kube-api-access-qwf8c\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.546346 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.546337 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvpj\" (UniqueName: \"kubernetes.io/projected/15066416-112b-4dfd-8761-2cf5116b8021-kube-api-access-9rvpj\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.584726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.557940 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:00.584726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.568940 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:00.912019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.911985 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7"] Nov 01 00:45:00.912409 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:00.912380 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee4f1a2_9040_4505_8001_5f5a23a2d122.slice/crio-365c7c09a2d820b90aebc538561e7bc94b839591794a286961fa0c9d8b84babd WatchSource:0}: Error finding container 365c7c09a2d820b90aebc538561e7bc94b839591794a286961fa0c9d8b84babd: Status 404 returned error can't find the container with id 365c7c09a2d820b90aebc538561e7bc94b839591794a286961fa0c9d8b84babd Nov 01 00:45:00.964058 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:00.964015 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95ed2b7_b0b6_4031_ab28_7d0cadddb621.slice/crio-55094a435f8db61ffb4cdeaa40a3c4c6affa01cb54194c79d2cf39a5a68e331d WatchSource:0}: Error finding container 55094a435f8db61ffb4cdeaa40a3c4c6affa01cb54194c79d2cf39a5a68e331d: Status 404 returned error can't find the container with id 55094a435f8db61ffb4cdeaa40a3c4c6affa01cb54194c79d2cf39a5a68e331d Nov 01 00:45:00.965691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.965431 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c"] Nov 01 00:45:00.966755 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:00.965952 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda98f1c_115c_4625_8f42_8850439f3008.slice/crio-73bec8113a12534e92d2e5f3819a728ac1b741a32ebd265a0c82f2299359a90a WatchSource:0}: Error finding container 73bec8113a12534e92d2e5f3819a728ac1b741a32ebd265a0c82f2299359a90a: Status 404 returned error can't find the container with id 73bec8113a12534e92d2e5f3819a728ac1b741a32ebd265a0c82f2299359a90a Nov 01 00:45:00.970985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:00.970478 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh"] Nov 01 00:45:01.015138 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:01.015114 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15066416_112b_4dfd_8761_2cf5116b8021.slice/crio-055e4ebfec6813961cf94a2c9347d41632587d041752c7ae1cef1ffc2a6784dd WatchSource:0}: Error finding container 055e4ebfec6813961cf94a2c9347d41632587d041752c7ae1cef1ffc2a6784dd: Status 404 returned error can't find the container with id 055e4ebfec6813961cf94a2c9347d41632587d041752c7ae1cef1ffc2a6784dd Nov 01 00:45:01.015983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.015955 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q"] Nov 01 00:45:01.021495 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.021474 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc"] Nov 01 00:45:01.021870 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:01.021850 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b5f06c_b6f6_4f13_b7ee_3f46090fbbef.slice/crio-187b7505331fee65a461b85e60df232ed4972807ae32997d40e349156a427a54 WatchSource:0}: Error finding container 187b7505331fee65a461b85e60df232ed4972807ae32997d40e349156a427a54: Status 404 returned error can't find the container with id 187b7505331fee65a461b85e60df232ed4972807ae32997d40e349156a427a54 Nov 01 00:45:01.589138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.589107 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerStarted","Data":"80b2c76948c02037dc703c60beac3a0ac24b43d9fee96304c9f7be9a0da3288c"} Nov 01 00:45:01.589582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.589200 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerStarted","Data":"187b7505331fee65a461b85e60df232ed4972807ae32997d40e349156a427a54"} Nov 01 00:45:01.590828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.590797 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerStarted","Data":"0e007b108e8f8921fb45b8301f084425336e5247503164c53d1956786af7547c"} Nov 01 00:45:01.590952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.590852 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerStarted","Data":"73bec8113a12534e92d2e5f3819a728ac1b741a32ebd265a0c82f2299359a90a"} Nov 01 00:45:01.592454 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.592430 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" event={"ID":"15066416-112b-4dfd-8761-2cf5116b8021","Type":"ContainerStarted","Data":"f15e11a4fabb5385e8c3822c8164c905e81f261e0d5e4e21833004203719fe5e"} Nov 01 00:45:01.592545 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.592470 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" event={"ID":"15066416-112b-4dfd-8761-2cf5116b8021","Type":"ContainerStarted","Data":"055e4ebfec6813961cf94a2c9347d41632587d041752c7ae1cef1ffc2a6784dd"} Nov 01 00:45:01.593922 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.593900 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerStarted","Data":"82c51e77430e0de2d75e084fe221e3160bf8510d80d0a7054ac7a1b4fcbaff38"} Nov 01 00:45:01.594024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.593944 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerStarted","Data":"365c7c09a2d820b90aebc538561e7bc94b839591794a286961fa0c9d8b84babd"} Nov 01 00:45:01.595772 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.595746 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerStarted","Data":"73a8842c99d8b8d61d75f63534fd0c3dc6b20563df9477aa7ae29635f42c0105"} Nov 01 00:45:01.596011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.595792 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerStarted","Data":"55094a435f8db61ffb4cdeaa40a3c4c6affa01cb54194c79d2cf39a5a68e331d"} Nov 01 00:45:01.707633 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.707568 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" podStartSLOduration=1.707555739 podStartE2EDuration="1.707555739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:01.7052849 +0000 UTC m=+1291.684867235" watchObservedRunningTime="2026-11-01 00:45:01.707555739 +0000 UTC m=+1291.687138071" Nov 01 00:45:01.824038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.823947 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" podStartSLOduration=1.823932264 podStartE2EDuration="1.823932264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:01.821568948 +0000 UTC m=+1291.801151301" watchObservedRunningTime="2026-11-01 00:45:01.823932264 +0000 UTC m=+1291.803514625" Nov 01 00:45:01.937859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:01.937791 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" podStartSLOduration=1.937772536 podStartE2EDuration="1.937772536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:01.936716377 +0000 UTC m=+1291.916298711" watchObservedRunningTime="2026-11-01 00:45:01.937772536 +0000 UTC m=+1291.917354871" Nov 01 00:45:02.056853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.056799 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" podStartSLOduration=2.056782528 podStartE2EDuration="2.056782528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:02.052669324 +0000 UTC m=+1292.032251659" watchObservedRunningTime="2026-11-01 00:45:02.056782528 +0000 UTC m=+1292.036364862" Nov 01 00:45:02.172810 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.172762 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" podStartSLOduration=2.172748675 podStartE2EDuration="2.172748675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:02.16990612 +0000 UTC m=+1292.149488456" watchObservedRunningTime="2026-11-01 00:45:02.172748675 +0000 UTC m=+1292.152331009" Nov 01 00:45:02.605092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.605053 4910 generic.go:356] "Generic (PLEG): container finished" podID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerID="80b2c76948c02037dc703c60beac3a0ac24b43d9fee96304c9f7be9a0da3288c" exitCode=60 Nov 01 00:45:02.605559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.605140 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerDied","Data":"80b2c76948c02037dc703c60beac3a0ac24b43d9fee96304c9f7be9a0da3288c"} Nov 01 00:45:02.605559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.605414 4910 scope.go:122] "RemoveContainer" containerID="80b2c76948c02037dc703c60beac3a0ac24b43d9fee96304c9f7be9a0da3288c" Nov 01 00:45:02.608240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.608208 4910 generic.go:356] "Generic (PLEG): container finished" podID="fda98f1c-115c-4625-8f42-8850439f3008" containerID="0e007b108e8f8921fb45b8301f084425336e5247503164c53d1956786af7547c" exitCode=60 Nov 01 00:45:02.608349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.608334 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerDied","Data":"0e007b108e8f8921fb45b8301f084425336e5247503164c53d1956786af7547c"} Nov 01 00:45:02.608689 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.608674 4910 scope.go:122] "RemoveContainer" containerID="0e007b108e8f8921fb45b8301f084425336e5247503164c53d1956786af7547c" Nov 01 00:45:02.611199 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.611177 4910 generic.go:356] "Generic (PLEG): container finished" podID="15066416-112b-4dfd-8761-2cf5116b8021" containerID="f15e11a4fabb5385e8c3822c8164c905e81f261e0d5e4e21833004203719fe5e" exitCode=60 Nov 01 00:45:02.611292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.611256 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" event={"ID":"15066416-112b-4dfd-8761-2cf5116b8021","Type":"ContainerDied","Data":"f15e11a4fabb5385e8c3822c8164c905e81f261e0d5e4e21833004203719fe5e"} Nov 01 00:45:02.611429 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.611415 4910 scope.go:122] "RemoveContainer" containerID="f15e11a4fabb5385e8c3822c8164c905e81f261e0d5e4e21833004203719fe5e" Nov 01 00:45:02.613095 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.613077 4910 generic.go:356] "Generic (PLEG): container finished" podID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerID="82c51e77430e0de2d75e084fe221e3160bf8510d80d0a7054ac7a1b4fcbaff38" exitCode=60 Nov 01 00:45:02.613192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.613163 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerDied","Data":"82c51e77430e0de2d75e084fe221e3160bf8510d80d0a7054ac7a1b4fcbaff38"} Nov 01 00:45:02.613353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.613338 4910 scope.go:122] "RemoveContainer" containerID="82c51e77430e0de2d75e084fe221e3160bf8510d80d0a7054ac7a1b4fcbaff38" Nov 01 00:45:02.614818 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.614796 4910 generic.go:356] "Generic (PLEG): container finished" podID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerID="73a8842c99d8b8d61d75f63534fd0c3dc6b20563df9477aa7ae29635f42c0105" exitCode=60 Nov 01 00:45:02.614940 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.614897 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerDied","Data":"73a8842c99d8b8d61d75f63534fd0c3dc6b20563df9477aa7ae29635f42c0105"} Nov 01 00:45:02.615098 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:02.615081 4910 scope.go:122] "RemoveContainer" containerID="73a8842c99d8b8d61d75f63534fd0c3dc6b20563df9477aa7ae29635f42c0105" Nov 01 00:45:03.624043 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.624014 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerStarted","Data":"8d15eef316d2527d076391ef6a95ca896587c8a3e829a2f7140e873b3a4100a0"} Nov 01 00:45:03.626334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.626305 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerStarted","Data":"36c1a536dd7c32a87d8260f8fe84353501983668a9e1e305c039a2e6b20fe1de"} Nov 01 00:45:03.628188 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.628163 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerStarted","Data":"3f1ca5a88872caed8d36f3cffb3582f7d238a5541e492763dee7f7a1a260d6ac"} Nov 01 00:45:03.637837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.637808 4910 generic.go:356] "Generic (PLEG): container finished" podID="15066416-112b-4dfd-8761-2cf5116b8021" containerID="338a7c9a62fd035d5ca8ba628262e437886e2650b586aac319f0bf7d6c0dcd0a" exitCode=60 Nov 01 00:45:03.637944 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.637889 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" event={"ID":"15066416-112b-4dfd-8761-2cf5116b8021","Type":"ContainerDied","Data":"338a7c9a62fd035d5ca8ba628262e437886e2650b586aac319f0bf7d6c0dcd0a"} Nov 01 00:45:03.638002 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.637967 4910 scope.go:122] "RemoveContainer" containerID="f15e11a4fabb5385e8c3822c8164c905e81f261e0d5e4e21833004203719fe5e" Nov 01 00:45:03.638193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.638139 4910 scope.go:122] "RemoveContainer" containerID="338a7c9a62fd035d5ca8ba628262e437886e2650b586aac319f0bf7d6c0dcd0a" Nov 01 00:45:03.638352 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:03.638334 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q_odh-ai-gateway-infra(15066416-112b-4dfd-8761-2cf5116b8021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" podUID="15066416-112b-4dfd-8761-2cf5116b8021" Nov 01 00:45:03.640227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:03.640201 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerStarted","Data":"1bd48023e4a9a96b17bb5c050b35104e45c976afb66c5e6be2c0530cc44dd4b4"} Nov 01 00:45:04.648723 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.648690 4910 generic.go:356] "Generic (PLEG): container finished" podID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerID="36c1a536dd7c32a87d8260f8fe84353501983668a9e1e305c039a2e6b20fe1de" exitCode=60 Nov 01 00:45:04.649228 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.648777 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerDied","Data":"36c1a536dd7c32a87d8260f8fe84353501983668a9e1e305c039a2e6b20fe1de"} Nov 01 00:45:04.649228 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.648850 4910 scope.go:122] "RemoveContainer" containerID="80b2c76948c02037dc703c60beac3a0ac24b43d9fee96304c9f7be9a0da3288c" Nov 01 00:45:04.649228 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.649021 4910 scope.go:122] "RemoveContainer" containerID="36c1a536dd7c32a87d8260f8fe84353501983668a9e1e305c039a2e6b20fe1de" Nov 01 00:45:04.649375 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:04.649309 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc_odh-ai-gateway-infra(77b5f06c-b6f6-4f13-b7ee-3f46090fbbef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" Nov 01 00:45:04.651535 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.651515 4910 generic.go:356] "Generic (PLEG): container finished" podID="fda98f1c-115c-4625-8f42-8850439f3008" containerID="3f1ca5a88872caed8d36f3cffb3582f7d238a5541e492763dee7f7a1a260d6ac" exitCode=60 Nov 01 00:45:04.651670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.651603 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerDied","Data":"3f1ca5a88872caed8d36f3cffb3582f7d238a5541e492763dee7f7a1a260d6ac"} Nov 01 00:45:04.651879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.651859 4910 scope.go:122] "RemoveContainer" containerID="3f1ca5a88872caed8d36f3cffb3582f7d238a5541e492763dee7f7a1a260d6ac" Nov 01 00:45:04.652092 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:04.652072 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh_odh-ai-gateway-infra(fda98f1c-115c-4625-8f42-8850439f3008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" podUID="fda98f1c-115c-4625-8f42-8850439f3008" Nov 01 00:45:04.653764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.653740 4910 scope.go:122] "RemoveContainer" containerID="338a7c9a62fd035d5ca8ba628262e437886e2650b586aac319f0bf7d6c0dcd0a" Nov 01 00:45:04.653983 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:04.653960 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q_odh-ai-gateway-infra(15066416-112b-4dfd-8761-2cf5116b8021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" podUID="15066416-112b-4dfd-8761-2cf5116b8021" Nov 01 00:45:04.656048 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.656023 4910 generic.go:356] "Generic (PLEG): container finished" podID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerID="1bd48023e4a9a96b17bb5c050b35104e45c976afb66c5e6be2c0530cc44dd4b4" exitCode=60 Nov 01 00:45:04.656175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.656060 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerDied","Data":"1bd48023e4a9a96b17bb5c050b35104e45c976afb66c5e6be2c0530cc44dd4b4"} Nov 01 00:45:04.656312 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.656294 4910 scope.go:122] "RemoveContainer" containerID="1bd48023e4a9a96b17bb5c050b35104e45c976afb66c5e6be2c0530cc44dd4b4" Nov 01 00:45:04.656548 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:04.656518 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7_odh-ai-gateway-infra(6ee4f1a2-9040-4505-8001-5f5a23a2d122)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" Nov 01 00:45:04.658378 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.658340 4910 generic.go:356] "Generic (PLEG): container finished" podID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerID="8d15eef316d2527d076391ef6a95ca896587c8a3e829a2f7140e873b3a4100a0" exitCode=60 Nov 01 00:45:04.658486 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.658445 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerDied","Data":"8d15eef316d2527d076391ef6a95ca896587c8a3e829a2f7140e873b3a4100a0"} Nov 01 00:45:04.658679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.658657 4910 scope.go:122] "RemoveContainer" containerID="8d15eef316d2527d076391ef6a95ca896587c8a3e829a2f7140e873b3a4100a0" Nov 01 00:45:04.658909 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:04.658888 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891565-nr76c_odh-ai-gateway-infra(d95ed2b7-b0b6-4031-ab28-7d0cadddb621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" Nov 01 00:45:04.664206 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.664185 4910 scope.go:122] "RemoveContainer" containerID="0e007b108e8f8921fb45b8301f084425336e5247503164c53d1956786af7547c" Nov 01 00:45:04.677556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.677536 4910 scope.go:122] "RemoveContainer" containerID="82c51e77430e0de2d75e084fe221e3160bf8510d80d0a7054ac7a1b4fcbaff38" Nov 01 00:45:04.692092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:04.692075 4910 scope.go:122] "RemoveContainer" containerID="73a8842c99d8b8d61d75f63534fd0c3dc6b20563df9477aa7ae29635f42c0105" Nov 01 00:45:05.669250 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:05.669205 4910 scope.go:122] "RemoveContainer" containerID="1bd48023e4a9a96b17bb5c050b35104e45c976afb66c5e6be2c0530cc44dd4b4" Nov 01 00:45:05.669796 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:05.669502 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7_odh-ai-gateway-infra(6ee4f1a2-9040-4505-8001-5f5a23a2d122)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" Nov 01 00:45:05.671113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:05.671092 4910 scope.go:122] "RemoveContainer" containerID="8d15eef316d2527d076391ef6a95ca896587c8a3e829a2f7140e873b3a4100a0" Nov 01 00:45:05.671357 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:05.671327 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891565-nr76c_odh-ai-gateway-infra(d95ed2b7-b0b6-4031-ab28-7d0cadddb621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" Nov 01 00:45:05.672922 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:05.672904 4910 scope.go:122] "RemoveContainer" containerID="36c1a536dd7c32a87d8260f8fe84353501983668a9e1e305c039a2e6b20fe1de" Nov 01 00:45:05.673161 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:05.673144 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc_odh-ai-gateway-infra(77b5f06c-b6f6-4f13-b7ee-3f46090fbbef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" Nov 01 00:45:05.674595 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:05.674580 4910 scope.go:122] "RemoveContainer" containerID="3f1ca5a88872caed8d36f3cffb3582f7d238a5541e492763dee7f7a1a260d6ac" Nov 01 00:45:05.674815 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:05.674798 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh_odh-ai-gateway-infra(fda98f1c-115c-4625-8f42-8850439f3008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" podUID="fda98f1c-115c-4625-8f42-8850439f3008" Nov 01 00:45:09.414259 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.414175 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b"] Nov 01 00:45:09.419806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.419788 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.475154 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.475115 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:45:09.480251 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.480214 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b"] Nov 01 00:45:09.496140 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.496109 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.496301 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.496168 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5qt\" (UniqueName: \"kubernetes.io/projected/88fcc7c7-109d-425c-a5c9-e33cba1c783e-kube-api-access-2f5qt\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.496301 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.496271 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.597602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.597566 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.597831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.597659 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5qt\" (UniqueName: \"kubernetes.io/projected/88fcc7c7-109d-425c-a5c9-e33cba1c783e-kube-api-access-2f5qt\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.597831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.597707 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.598438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.598414 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.600087 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.600064 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.659069 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.659027 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5qt\" (UniqueName: \"kubernetes.io/projected/88fcc7c7-109d-425c-a5c9-e33cba1c783e-kube-api-access-2f5qt\") pod \"maas-api-revoke-sub-33ce4550-5zg4b\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:09.733223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:09.733147 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:10.112670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:10.112643 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b"] Nov 01 00:45:10.113323 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:10.113297 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fcc7c7_109d_425c_a5c9_e33cba1c783e.slice/crio-34e30cb5702e3d5d6112e59ce111f7addd4904f313a7f778932b9ab9b96da36f WatchSource:0}: Error finding container 34e30cb5702e3d5d6112e59ce111f7addd4904f313a7f778932b9ab9b96da36f: Status 404 returned error can't find the container with id 34e30cb5702e3d5d6112e59ce111f7addd4904f313a7f778932b9ab9b96da36f Nov 01 00:45:10.710098 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:10.710019 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" event={"ID":"88fcc7c7-109d-425c-a5c9-e33cba1c783e","Type":"ContainerStarted","Data":"84ce11e89e9459255584767bb614052aa8bf54c260ec8a2ddbeb9d8e46d285cb"} Nov 01 00:45:10.710098 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:10.710077 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" event={"ID":"88fcc7c7-109d-425c-a5c9-e33cba1c783e","Type":"ContainerStarted","Data":"34e30cb5702e3d5d6112e59ce111f7addd4904f313a7f778932b9ab9b96da36f"} Nov 01 00:45:10.877871 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:10.877808 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" podStartSLOduration=1.877788544 podStartE2EDuration="1.877788544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:10.825779085 +0000 UTC m=+1300.805361421" watchObservedRunningTime="2026-11-01 00:45:10.877788544 +0000 UTC m=+1300.857370880" Nov 01 00:45:11.717678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:11.717616 4910 generic.go:356] "Generic (PLEG): container finished" podID="88fcc7c7-109d-425c-a5c9-e33cba1c783e" containerID="84ce11e89e9459255584767bb614052aa8bf54c260ec8a2ddbeb9d8e46d285cb" exitCode=0 Nov 01 00:45:11.718122 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:11.717732 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" event={"ID":"88fcc7c7-109d-425c-a5c9-e33cba1c783e","Type":"ContainerDied","Data":"84ce11e89e9459255584767bb614052aa8bf54c260ec8a2ddbeb9d8e46d285cb"} Nov 01 00:45:12.886826 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.886799 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:12.932718 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.932692 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-xks-ca\") pod \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " Nov 01 00:45:12.932935 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.932768 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fcc7c7-109d-425c-a5c9-e33cba1c783e-kube-api-access-2f5qt\" (UniqueName: \"kubernetes.io/projected/88fcc7c7-109d-425c-a5c9-e33cba1c783e-kube-api-access-2f5qt\") pod \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " Nov 01 00:45:12.932935 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.932797 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-openshift-ca\") pod \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\" (UID: \"88fcc7c7-109d-425c-a5c9-e33cba1c783e\") " Nov 01 00:45:12.933144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.933118 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-openshift-ca" pod "88fcc7c7-109d-425c-a5c9-e33cba1c783e" (UID: "88fcc7c7-109d-425c-a5c9-e33cba1c783e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:12.933288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.933271 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:12.934870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.934849 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-xks-ca" pod "88fcc7c7-109d-425c-a5c9-e33cba1c783e" (UID: "88fcc7c7-109d-425c-a5c9-e33cba1c783e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:12.934944 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:12.934860 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fcc7c7-109d-425c-a5c9-e33cba1c783e-kube-api-access-2f5qt" pod "88fcc7c7-109d-425c-a5c9-e33cba1c783e" (UID: "88fcc7c7-109d-425c-a5c9-e33cba1c783e"). InnerVolumeSpecName "kube-api-access-2f5qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:13.034449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:13.034415 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88fcc7c7-109d-425c-a5c9-e33cba1c783e-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:13.034602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:13.034454 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5qt\" (UniqueName: \"kubernetes.io/projected/88fcc7c7-109d-425c-a5c9-e33cba1c783e-kube-api-access-2f5qt\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:13.104151 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:13.104108 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:45:13.739755 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:13.739718 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" Nov 01 00:45:13.739909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:13.739752 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b" event={"ID":"88fcc7c7-109d-425c-a5c9-e33cba1c783e","Type":"ContainerDied","Data":"34e30cb5702e3d5d6112e59ce111f7addd4904f313a7f778932b9ab9b96da36f"} Nov 01 00:45:13.739909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:13.739830 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e30cb5702e3d5d6112e59ce111f7addd4904f313a7f778932b9ab9b96da36f" Nov 01 00:45:14.193759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:14.193721 4910 scope.go:122] "RemoveContainer" containerID="1bd48023e4a9a96b17bb5c050b35104e45c976afb66c5e6be2c0530cc44dd4b4" Nov 01 00:45:14.748391 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:14.748360 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerStarted","Data":"963f6c68c6f052ed06b178a546a7af94f489d0d424988cec7f697ae5ecc5eba9"} Nov 01 00:45:15.184657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:15.184593 4910 scope.go:122] "RemoveContainer" containerID="3f1ca5a88872caed8d36f3cffb3582f7d238a5541e492763dee7f7a1a260d6ac" Nov 01 00:45:15.759217 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:15.759182 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerStarted","Data":"4485b1d68b2906d0496aa52cc4ed7e53d5e3286268b800c7fb06aa656e26ea40"} Nov 01 00:45:15.761548 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:15.761518 4910 generic.go:356] "Generic (PLEG): container finished" podID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerID="963f6c68c6f052ed06b178a546a7af94f489d0d424988cec7f697ae5ecc5eba9" exitCode=60 Nov 01 00:45:15.761548 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:15.761533 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerDied","Data":"963f6c68c6f052ed06b178a546a7af94f489d0d424988cec7f697ae5ecc5eba9"} Nov 01 00:45:15.761793 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:15.761601 4910 scope.go:122] "RemoveContainer" containerID="1bd48023e4a9a96b17bb5c050b35104e45c976afb66c5e6be2c0530cc44dd4b4" Nov 01 00:45:15.761955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:15.761930 4910 scope.go:122] "RemoveContainer" containerID="963f6c68c6f052ed06b178a546a7af94f489d0d424988cec7f697ae5ecc5eba9" Nov 01 00:45:15.762202 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:15.762184 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7_odh-ai-gateway-infra(6ee4f1a2-9040-4505-8001-5f5a23a2d122)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" Nov 01 00:45:15.874519 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:15.874491 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7"] Nov 01 00:45:16.027727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.027656 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9"] Nov 01 00:45:16.028177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.028165 4910 cpu_manager.go:414] "containerMap: removing container" podUID="88fcc7c7-109d-425c-a5c9-e33cba1c783e" containerName="revoke-keys" Nov 01 00:45:16.028223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.028179 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fcc7c7-109d-425c-a5c9-e33cba1c783e" containerName="revoke-keys" Nov 01 00:45:16.028294 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.028285 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fcc7c7-109d-425c-a5c9-e33cba1c783e" containerName="revoke-keys" Nov 01 00:45:16.032939 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.032919 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.066175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.066154 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.066292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.066191 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56kx\" (UniqueName: \"kubernetes.io/projected/bd2b84cb-54c9-4b22-9118-a0e379c70fde-kube-api-access-n56kx\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.066292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.066214 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.086280 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.086255 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:45:16.090200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.090179 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9"] Nov 01 00:45:16.168417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.168376 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.168674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.168446 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56kx\" (UniqueName: \"kubernetes.io/projected/bd2b84cb-54c9-4b22-9118-a0e379c70fde-kube-api-access-n56kx\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.168674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.168479 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.169449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.169422 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.177088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.177064 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.191897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.191878 4910 scope.go:122] "RemoveContainer" containerID="8d15eef316d2527d076391ef6a95ca896587c8a3e829a2f7140e873b3a4100a0" Nov 01 00:45:16.229399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.229382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56kx\" (UniqueName: \"kubernetes.io/projected/bd2b84cb-54c9-4b22-9118-a0e379c70fde-kube-api-access-n56kx\") pod \"maas-api-revoke-sub-63d8d10c-52jk9\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.346513 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.346441 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:16.728894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.728868 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9"] Nov 01 00:45:16.729509 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:16.729483 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2b84cb_54c9_4b22_9118_a0e379c70fde.slice/crio-e48ffd9d68e08ed92731436b9ab14a2619e1f05cf49928377fde23244554bb7b WatchSource:0}: Error finding container e48ffd9d68e08ed92731436b9ab14a2619e1f05cf49928377fde23244554bb7b: Status 404 returned error can't find the container with id e48ffd9d68e08ed92731436b9ab14a2619e1f05cf49928377fde23244554bb7b Nov 01 00:45:16.771955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.771916 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" event={"ID":"bd2b84cb-54c9-4b22-9118-a0e379c70fde","Type":"ContainerStarted","Data":"e48ffd9d68e08ed92731436b9ab14a2619e1f05cf49928377fde23244554bb7b"} Nov 01 00:45:16.773965 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.773942 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerStarted","Data":"2aa448431cc7f6bead79f7ef4a762ad082ba14215add07099fc79486cff8353b"} Nov 01 00:45:16.776046 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.776023 4910 generic.go:356] "Generic (PLEG): container finished" podID="fda98f1c-115c-4625-8f42-8850439f3008" containerID="4485b1d68b2906d0496aa52cc4ed7e53d5e3286268b800c7fb06aa656e26ea40" exitCode=60 Nov 01 00:45:16.776137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.776055 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerDied","Data":"4485b1d68b2906d0496aa52cc4ed7e53d5e3286268b800c7fb06aa656e26ea40"} Nov 01 00:45:16.776137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.776108 4910 scope.go:122] "RemoveContainer" containerID="3f1ca5a88872caed8d36f3cffb3582f7d238a5541e492763dee7f7a1a260d6ac" Nov 01 00:45:16.776367 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.776352 4910 scope.go:122] "RemoveContainer" containerID="4485b1d68b2906d0496aa52cc4ed7e53d5e3286268b800c7fb06aa656e26ea40" Nov 01 00:45:16.776614 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:16.776594 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh_odh-ai-gateway-infra(fda98f1c-115c-4625-8f42-8850439f3008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" podUID="fda98f1c-115c-4625-8f42-8850439f3008" Nov 01 00:45:16.925830 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.925799 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh"] Nov 01 00:45:16.939267 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.939245 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:16.996904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.996859 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee4f1a2-9040-4505-8001-5f5a23a2d122-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee4f1a2-9040-4505-8001-5f5a23a2d122-maas-api-service-ca\") pod \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " Nov 01 00:45:16.997108 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.996928 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee4f1a2-9040-4505-8001-5f5a23a2d122-kube-api-access-22mn9\" (UniqueName: \"kubernetes.io/projected/6ee4f1a2-9040-4505-8001-5f5a23a2d122-kube-api-access-22mn9\") pod \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\" (UID: \"6ee4f1a2-9040-4505-8001-5f5a23a2d122\") " Nov 01 00:45:16.997297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.997271 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee4f1a2-9040-4505-8001-5f5a23a2d122-maas-api-service-ca" pod "6ee4f1a2-9040-4505-8001-5f5a23a2d122" (UID: "6ee4f1a2-9040-4505-8001-5f5a23a2d122"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:16.997444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.997417 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ee4f1a2-9040-4505-8001-5f5a23a2d122-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:16.999661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:16.999604 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee4f1a2-9040-4505-8001-5f5a23a2d122-kube-api-access-22mn9" pod "6ee4f1a2-9040-4505-8001-5f5a23a2d122" (UID: "6ee4f1a2-9040-4505-8001-5f5a23a2d122"). InnerVolumeSpecName "kube-api-access-22mn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:17.099207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.099174 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22mn9\" (UniqueName: \"kubernetes.io/projected/6ee4f1a2-9040-4505-8001-5f5a23a2d122-kube-api-access-22mn9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:17.184285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.184247 4910 scope.go:122] "RemoveContainer" containerID="36c1a536dd7c32a87d8260f8fe84353501983668a9e1e305c039a2e6b20fe1de" Nov 01 00:45:17.792930 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.792794 4910 generic.go:356] "Generic (PLEG): container finished" podID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerID="2aa448431cc7f6bead79f7ef4a762ad082ba14215add07099fc79486cff8353b" exitCode=60 Nov 01 00:45:17.792930 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.792876 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerDied","Data":"2aa448431cc7f6bead79f7ef4a762ad082ba14215add07099fc79486cff8353b"} Nov 01 00:45:17.793759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.792961 4910 scope.go:122] "RemoveContainer" containerID="8d15eef316d2527d076391ef6a95ca896587c8a3e829a2f7140e873b3a4100a0" Nov 01 00:45:17.793759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.793408 4910 scope.go:122] "RemoveContainer" containerID="2aa448431cc7f6bead79f7ef4a762ad082ba14215add07099fc79486cff8353b" Nov 01 00:45:17.794670 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:17.793829 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891565-nr76c_odh-ai-gateway-infra(d95ed2b7-b0b6-4031-ab28-7d0cadddb621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" Nov 01 00:45:17.798385 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.798334 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerStarted","Data":"7269da7f494b4e78b05d4f04ea394aec7a3f369d391eb4b87a40b49f5afc92ca"} Nov 01 00:45:17.804029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.804003 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" event={"ID":"6ee4f1a2-9040-4505-8001-5f5a23a2d122","Type":"ContainerDied","Data":"365c7c09a2d820b90aebc538561e7bc94b839591794a286961fa0c9d8b84babd"} Nov 01 00:45:17.804121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.804093 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7" Nov 01 00:45:17.815588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.815564 4910 scope.go:122] "RemoveContainer" containerID="963f6c68c6f052ed06b178a546a7af94f489d0d424988cec7f697ae5ecc5eba9" Nov 01 00:45:17.819890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:17.819859 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" event={"ID":"bd2b84cb-54c9-4b22-9118-a0e379c70fde","Type":"ContainerStarted","Data":"41f2b45ec37a8153a18a60a3eb0b12a21709bea8d5cf567df7d072c948c642a3"} Nov 01 00:45:18.025708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.025682 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c"] Nov 01 00:45:18.029573 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.029549 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7"] Nov 01 00:45:18.036860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.036828 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-22d7b2-29891565-vrqs7"] Nov 01 00:45:18.077857 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.077826 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:18.184910 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.184883 4910 scope.go:122] "RemoveContainer" containerID="338a7c9a62fd035d5ca8ba628262e437886e2650b586aac319f0bf7d6c0dcd0a" Nov 01 00:45:18.192588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.192562 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" path="/var/lib/kubelet/pods/6ee4f1a2-9040-4505-8001-5f5a23a2d122/volumes" Nov 01 00:45:18.224958 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.224931 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda98f1c-115c-4625-8f42-8850439f3008-kube-api-access-7cdlf\" (UniqueName: \"kubernetes.io/projected/fda98f1c-115c-4625-8f42-8850439f3008-kube-api-access-7cdlf\") pod \"fda98f1c-115c-4625-8f42-8850439f3008\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " Nov 01 00:45:18.225117 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.225102 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda98f1c-115c-4625-8f42-8850439f3008-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda98f1c-115c-4625-8f42-8850439f3008-maas-api-service-ca\") pod \"fda98f1c-115c-4625-8f42-8850439f3008\" (UID: \"fda98f1c-115c-4625-8f42-8850439f3008\") " Nov 01 00:45:18.225440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.225412 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda98f1c-115c-4625-8f42-8850439f3008-maas-api-service-ca" pod "fda98f1c-115c-4625-8f42-8850439f3008" (UID: "fda98f1c-115c-4625-8f42-8850439f3008"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:18.227071 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.227055 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda98f1c-115c-4625-8f42-8850439f3008-kube-api-access-7cdlf" pod "fda98f1c-115c-4625-8f42-8850439f3008" (UID: "fda98f1c-115c-4625-8f42-8850439f3008"). InnerVolumeSpecName "kube-api-access-7cdlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:18.326882 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.326571 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cdlf\" (UniqueName: \"kubernetes.io/projected/fda98f1c-115c-4625-8f42-8850439f3008-kube-api-access-7cdlf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:18.326882 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.326607 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda98f1c-115c-4625-8f42-8850439f3008-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:18.385866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.385804 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" podStartSLOduration=3.385784525 podStartE2EDuration="3.385784525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:18.379984655 +0000 UTC m=+1308.359567015" watchObservedRunningTime="2026-11-01 00:45:18.385784525 +0000 UTC m=+1308.365366861" Nov 01 00:45:18.829020 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.828984 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" Nov 01 00:45:18.829020 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.829010 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh" event={"ID":"fda98f1c-115c-4625-8f42-8850439f3008","Type":"ContainerDied","Data":"73bec8113a12534e92d2e5f3819a728ac1b741a32ebd265a0c82f2299359a90a"} Nov 01 00:45:18.829560 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.829087 4910 scope.go:122] "RemoveContainer" containerID="4485b1d68b2906d0496aa52cc4ed7e53d5e3286268b800c7fb06aa656e26ea40" Nov 01 00:45:18.831451 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.831419 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" event={"ID":"15066416-112b-4dfd-8761-2cf5116b8021","Type":"ContainerStarted","Data":"fc1a8ff4bd16ca60963e1dd9b61d5d120fbe77883ce2a4a98d79f9199832d3f2"} Nov 01 00:45:18.834405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.834378 4910 generic.go:356] "Generic (PLEG): container finished" podID="bd2b84cb-54c9-4b22-9118-a0e379c70fde" containerID="41f2b45ec37a8153a18a60a3eb0b12a21709bea8d5cf567df7d072c948c642a3" exitCode=0 Nov 01 00:45:18.834551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.834455 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" event={"ID":"bd2b84cb-54c9-4b22-9118-a0e379c70fde","Type":"ContainerDied","Data":"41f2b45ec37a8153a18a60a3eb0b12a21709bea8d5cf567df7d072c948c642a3"} Nov 01 00:45:18.839212 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.839189 4910 generic.go:356] "Generic (PLEG): container finished" podID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerID="7269da7f494b4e78b05d4f04ea394aec7a3f369d391eb4b87a40b49f5afc92ca" exitCode=60 Nov 01 00:45:18.839318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.839222 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerDied","Data":"7269da7f494b4e78b05d4f04ea394aec7a3f369d391eb4b87a40b49f5afc92ca"} Nov 01 00:45:18.839660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.839536 4910 scope.go:122] "RemoveContainer" containerID="7269da7f494b4e78b05d4f04ea394aec7a3f369d391eb4b87a40b49f5afc92ca" Nov 01 00:45:18.839866 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:18.839789 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc_odh-ai-gateway-infra(77b5f06c-b6f6-4f13-b7ee-3f46090fbbef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" Nov 01 00:45:18.841866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:18.841850 4910 scope.go:122] "RemoveContainer" containerID="36c1a536dd7c32a87d8260f8fe84353501983668a9e1e305c039a2e6b20fe1de" Nov 01 00:45:19.015637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.015608 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:19.134504 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.134437 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-kube-api-access-svjhv\" (UniqueName: \"kubernetes.io/projected/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-kube-api-access-svjhv\") pod \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " Nov 01 00:45:19.134686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.134525 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-maas-api-service-ca\") pod \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\" (UID: \"d95ed2b7-b0b6-4031-ab28-7d0cadddb621\") " Nov 01 00:45:19.134880 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.134853 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-maas-api-service-ca" pod "d95ed2b7-b0b6-4031-ab28-7d0cadddb621" (UID: "d95ed2b7-b0b6-4031-ab28-7d0cadddb621"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:19.135026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.134951 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:19.136756 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.136737 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-kube-api-access-svjhv" pod "d95ed2b7-b0b6-4031-ab28-7d0cadddb621" (UID: "d95ed2b7-b0b6-4031-ab28-7d0cadddb621"). InnerVolumeSpecName "kube-api-access-svjhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:19.163536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.163507 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc"] Nov 01 00:45:19.240430 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.240399 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svjhv\" (UniqueName: \"kubernetes.io/projected/d95ed2b7-b0b6-4031-ab28-7d0cadddb621-kube-api-access-svjhv\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:19.342234 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.342203 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh"] Nov 01 00:45:19.346751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.346727 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-32439f-29891565-b9pjh"] Nov 01 00:45:19.848532 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.848503 4910 generic.go:356] "Generic (PLEG): container finished" podID="15066416-112b-4dfd-8761-2cf5116b8021" containerID="fc1a8ff4bd16ca60963e1dd9b61d5d120fbe77883ce2a4a98d79f9199832d3f2" exitCode=60 Nov 01 00:45:19.849046 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.848573 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" event={"ID":"15066416-112b-4dfd-8761-2cf5116b8021","Type":"ContainerDied","Data":"fc1a8ff4bd16ca60963e1dd9b61d5d120fbe77883ce2a4a98d79f9199832d3f2"} Nov 01 00:45:19.849046 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.848686 4910 scope.go:122] "RemoveContainer" containerID="338a7c9a62fd035d5ca8ba628262e437886e2650b586aac319f0bf7d6c0dcd0a" Nov 01 00:45:19.849046 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.849035 4910 scope.go:122] "RemoveContainer" containerID="fc1a8ff4bd16ca60963e1dd9b61d5d120fbe77883ce2a4a98d79f9199832d3f2" Nov 01 00:45:19.849361 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:19.849332 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q_odh-ai-gateway-infra(15066416-112b-4dfd-8761-2cf5116b8021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" podUID="15066416-112b-4dfd-8761-2cf5116b8021" Nov 01 00:45:19.850690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.850670 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" Nov 01 00:45:19.850785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.850696 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c" event={"ID":"d95ed2b7-b0b6-4031-ab28-7d0cadddb621","Type":"ContainerDied","Data":"55094a435f8db61ffb4cdeaa40a3c4c6affa01cb54194c79d2cf39a5a68e331d"} Nov 01 00:45:19.869847 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:19.868945 4910 scope.go:122] "RemoveContainer" containerID="2aa448431cc7f6bead79f7ef4a762ad082ba14215add07099fc79486cff8353b" Nov 01 00:45:20.048931 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.048902 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:20.065840 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.065725 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:20.080023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.079999 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c"] Nov 01 00:45:20.086201 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.086177 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891565-nr76c"] Nov 01 00:45:20.122384 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.122329 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q"] Nov 01 00:45:20.148923 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.148900 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-openshift-ca\") pod \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " Nov 01 00:45:20.149044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.148943 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2b84cb-54c9-4b22-9118-a0e379c70fde-kube-api-access-n56kx\" (UniqueName: \"kubernetes.io/projected/bd2b84cb-54c9-4b22-9118-a0e379c70fde-kube-api-access-n56kx\") pod \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " Nov 01 00:45:20.149044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.149036 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-xks-ca\") pod \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\" (UID: \"bd2b84cb-54c9-4b22-9118-a0e379c70fde\") " Nov 01 00:45:20.149161 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.149103 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-maas-api-service-ca\") pod \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " Nov 01 00:45:20.149161 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.149134 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-kube-api-access-qwf8c\" (UniqueName: \"kubernetes.io/projected/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-kube-api-access-qwf8c\") pod \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\" (UID: \"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef\") " Nov 01 00:45:20.149483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.149331 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-openshift-ca" pod "bd2b84cb-54c9-4b22-9118-a0e379c70fde" (UID: "bd2b84cb-54c9-4b22-9118-a0e379c70fde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:20.149483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.149441 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:20.149616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.149558 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-maas-api-service-ca" pod "77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" (UID: "77b5f06c-b6f6-4f13-b7ee-3f46090fbbef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:20.152011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.151990 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2b84cb-54c9-4b22-9118-a0e379c70fde-kube-api-access-n56kx" pod "bd2b84cb-54c9-4b22-9118-a0e379c70fde" (UID: "bd2b84cb-54c9-4b22-9118-a0e379c70fde"). InnerVolumeSpecName "kube-api-access-n56kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:20.152582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.152557 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-xks-ca" pod "bd2b84cb-54c9-4b22-9118-a0e379c70fde" (UID: "bd2b84cb-54c9-4b22-9118-a0e379c70fde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:20.152858 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.152829 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-kube-api-access-qwf8c" pod "77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" (UID: "77b5f06c-b6f6-4f13-b7ee-3f46090fbbef"). InnerVolumeSpecName "kube-api-access-qwf8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:20.191828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.191804 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" path="/var/lib/kubelet/pods/d95ed2b7-b0b6-4031-ab28-7d0cadddb621/volumes" Nov 01 00:45:20.192194 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.192181 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda98f1c-115c-4625-8f42-8850439f3008" path="/var/lib/kubelet/pods/fda98f1c-115c-4625-8f42-8850439f3008/volumes" Nov 01 00:45:20.250213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.250181 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b84cb-54c9-4b22-9118-a0e379c70fde-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:20.250213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.250214 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:20.250432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.250229 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwf8c\" (UniqueName: \"kubernetes.io/projected/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef-kube-api-access-qwf8c\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:20.250432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.250243 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56kx\" (UniqueName: \"kubernetes.io/projected/bd2b84cb-54c9-4b22-9118-a0e379c70fde-kube-api-access-n56kx\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:20.866311 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.866275 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" Nov 01 00:45:20.866860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.866308 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc" event={"ID":"77b5f06c-b6f6-4f13-b7ee-3f46090fbbef","Type":"ContainerDied","Data":"187b7505331fee65a461b85e60df232ed4972807ae32997d40e349156a427a54"} Nov 01 00:45:20.866860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.866385 4910 scope.go:122] "RemoveContainer" containerID="7269da7f494b4e78b05d4f04ea394aec7a3f369d391eb4b87a40b49f5afc92ca" Nov 01 00:45:20.872459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.872422 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" Nov 01 00:45:20.872459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.872430 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9" event={"ID":"bd2b84cb-54c9-4b22-9118-a0e379c70fde","Type":"ContainerDied","Data":"e48ffd9d68e08ed92731436b9ab14a2619e1f05cf49928377fde23244554bb7b"} Nov 01 00:45:20.872688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:20.872476 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48ffd9d68e08ed92731436b9ab14a2619e1f05cf49928377fde23244554bb7b" Nov 01 00:45:21.022950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.022925 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:21.158784 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.158700 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc"] Nov 01 00:45:21.160236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.160214 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15066416-112b-4dfd-8761-2cf5116b8021-kube-api-access-9rvpj\" (UniqueName: \"kubernetes.io/projected/15066416-112b-4dfd-8761-2cf5116b8021-kube-api-access-9rvpj\") pod \"15066416-112b-4dfd-8761-2cf5116b8021\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " Nov 01 00:45:21.160451 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.160433 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15066416-112b-4dfd-8761-2cf5116b8021-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15066416-112b-4dfd-8761-2cf5116b8021-maas-api-service-ca\") pod \"15066416-112b-4dfd-8761-2cf5116b8021\" (UID: \"15066416-112b-4dfd-8761-2cf5116b8021\") " Nov 01 00:45:21.160898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.160866 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15066416-112b-4dfd-8761-2cf5116b8021-maas-api-service-ca" pod "15066416-112b-4dfd-8761-2cf5116b8021" (UID: "15066416-112b-4dfd-8761-2cf5116b8021"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:21.162588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.162551 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15066416-112b-4dfd-8761-2cf5116b8021-kube-api-access-9rvpj" pod "15066416-112b-4dfd-8761-2cf5116b8021" (UID: "15066416-112b-4dfd-8761-2cf5116b8021"). InnerVolumeSpecName "kube-api-access-9rvpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:21.171490 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.171467 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891565-gztbc"] Nov 01 00:45:21.261902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.261867 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvpj\" (UniqueName: \"kubernetes.io/projected/15066416-112b-4dfd-8761-2cf5116b8021-kube-api-access-9rvpj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:21.261902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.261902 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15066416-112b-4dfd-8761-2cf5116b8021-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:21.883475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.883450 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" Nov 01 00:45:21.883475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.883467 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q" event={"ID":"15066416-112b-4dfd-8761-2cf5116b8021","Type":"ContainerDied","Data":"055e4ebfec6813961cf94a2c9347d41632587d041752c7ae1cef1ffc2a6784dd"} Nov 01 00:45:21.884064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:21.883529 4910 scope.go:122] "RemoveContainer" containerID="fc1a8ff4bd16ca60963e1dd9b61d5d120fbe77883ce2a4a98d79f9199832d3f2" Nov 01 00:45:22.050244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.050197 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q"] Nov 01 00:45:22.054036 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.054010 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891565-v7r2q"] Nov 01 00:45:22.195083 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.194995 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15066416-112b-4dfd-8761-2cf5116b8021" path="/var/lib/kubelet/pods/15066416-112b-4dfd-8761-2cf5116b8021/volumes" Nov 01 00:45:22.195538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.195496 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" path="/var/lib/kubelet/pods/77b5f06c-b6f6-4f13-b7ee-3f46090fbbef/volumes" Nov 01 00:45:22.602499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.602475 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k"] Nov 01 00:45:22.603088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603072 4910 cpu_manager.go:414] "containerMap: removing container" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603091 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603100 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603106 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603113 4910 cpu_manager.go:414] "containerMap: removing container" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603118 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603127 4910 cpu_manager.go:414] "containerMap: removing container" podUID="bd2b84cb-54c9-4b22-9118-a0e379c70fde" containerName="revoke-keys" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603131 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2b84cb-54c9-4b22-9118-a0e379c70fde" containerName="revoke-keys" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603141 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:22.603153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603146 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603158 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603163 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603173 4910 cpu_manager.go:414] "containerMap: removing container" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603177 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603185 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603189 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603195 4910 cpu_manager.go:414] "containerMap: removing container" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603199 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603208 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603212 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603218 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603223 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603233 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603238 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603319 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603327 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603333 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603339 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603344 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603349 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603354 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603358 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603364 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603368 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603373 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603378 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:22.603413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.603384 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2b84cb-54c9-4b22-9118-a0e379c70fde" containerName="revoke-keys" Nov 01 00:45:22.606704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.606687 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.659354 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.659323 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:45:22.659564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.659546 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:45:22.661234 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.661205 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k"] Nov 01 00:45:22.679427 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.679389 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.679535 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.679473 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.679611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.679579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kzb\" (UniqueName: \"kubernetes.io/projected/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-kube-api-access-h2kzb\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.781170 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.781121 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kzb\" (UniqueName: \"kubernetes.io/projected/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-kube-api-access-h2kzb\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.781423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.781230 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.781423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.781293 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.782080 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.782053 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.783849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.783829 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.842681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.842601 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kzb\" (UniqueName: \"kubernetes.io/projected/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-kube-api-access-h2kzb\") pod \"maas-api-revoke-sub-c3156970-msj4k\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:22.894530 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.894502 4910 generic.go:356] "Generic (PLEG): container finished" podID="14899c7c-f079-4e5a-b044-9cc72f305b83" containerID="6e311a980737a5f7b9f0777e7c795fe3f33e9f0b7cf27d81b2fdac89e69c98b1" exitCode=0 Nov 01 00:45:22.894901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.894570 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" event={"ID":"14899c7c-f079-4e5a-b044-9cc72f305b83","Type":"ContainerDied","Data":"6e311a980737a5f7b9f0777e7c795fe3f33e9f0b7cf27d81b2fdac89e69c98b1"} Nov 01 00:45:22.983814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:22.983785 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:23.368171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:23.368141 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k"] Nov 01 00:45:23.368406 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:23.368381 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5b18f4_2e9c_46f0_b439_d95b6e4161ef.slice/crio-9b52bca198d6a93ff740bbf91cc557620ca1713da6bb8768bc922a3aa2fd55c7 WatchSource:0}: Error finding container 9b52bca198d6a93ff740bbf91cc557620ca1713da6bb8768bc922a3aa2fd55c7: Status 404 returned error can't find the container with id 9b52bca198d6a93ff740bbf91cc557620ca1713da6bb8768bc922a3aa2fd55c7 Nov 01 00:45:23.903003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:23.902959 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" event={"ID":"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef","Type":"ContainerStarted","Data":"573eaef2a18661115b7b55d9fb7c025c2f0b8fea9d48000fecdfe25c904fa154"} Nov 01 00:45:23.903374 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:23.903018 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" event={"ID":"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef","Type":"ContainerStarted","Data":"9b52bca198d6a93ff740bbf91cc557620ca1713da6bb8768bc922a3aa2fd55c7"} Nov 01 00:45:24.023076 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.023020 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" podStartSLOduration=2.023001661 podStartE2EDuration="2.023001661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:24.020489821 +0000 UTC m=+1314.000072156" watchObservedRunningTime="2026-11-01 00:45:24.023001661 +0000 UTC m=+1314.002584000" Nov 01 00:45:24.056920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.056899 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:45:24.196978 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.196933 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14899c7c-f079-4e5a-b044-9cc72f305b83-kube-api-access-c5cq2\" (UniqueName: \"kubernetes.io/projected/14899c7c-f079-4e5a-b044-9cc72f305b83-kube-api-access-c5cq2\") pod \"14899c7c-f079-4e5a-b044-9cc72f305b83\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " Nov 01 00:45:24.197169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.197050 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14899c7c-f079-4e5a-b044-9cc72f305b83-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/14899c7c-f079-4e5a-b044-9cc72f305b83-archives-path\") pod \"14899c7c-f079-4e5a-b044-9cc72f305b83\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " Nov 01 00:45:24.197169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.197129 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-service-ca-bundle\") pod \"14899c7c-f079-4e5a-b044-9cc72f305b83\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " Nov 01 00:45:24.197306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.197173 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-trusted-ca-bundle\") pod \"14899c7c-f079-4e5a-b044-9cc72f305b83\" (UID: \"14899c7c-f079-4e5a-b044-9cc72f305b83\") " Nov 01 00:45:24.197727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.197697 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14899c7c-f079-4e5a-b044-9cc72f305b83-archives-path" pod "14899c7c-f079-4e5a-b044-9cc72f305b83" (UID: "14899c7c-f079-4e5a-b044-9cc72f305b83"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:45:24.197852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.197776 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-trusted-ca-bundle" pod "14899c7c-f079-4e5a-b044-9cc72f305b83" (UID: "14899c7c-f079-4e5a-b044-9cc72f305b83"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:24.198125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.198096 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-service-ca-bundle" pod "14899c7c-f079-4e5a-b044-9cc72f305b83" (UID: "14899c7c-f079-4e5a-b044-9cc72f305b83"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:24.200271 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.200245 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14899c7c-f079-4e5a-b044-9cc72f305b83-kube-api-access-c5cq2" pod "14899c7c-f079-4e5a-b044-9cc72f305b83" (UID: "14899c7c-f079-4e5a-b044-9cc72f305b83"). InnerVolumeSpecName "kube-api-access-c5cq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:24.298549 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.298522 4910 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-service-ca-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:24.298549 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.298546 4910 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14899c7c-f079-4e5a-b044-9cc72f305b83-trusted-ca-bundle\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:24.298775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.298556 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5cq2\" (UniqueName: \"kubernetes.io/projected/14899c7c-f079-4e5a-b044-9cc72f305b83-kube-api-access-c5cq2\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:24.298775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.298567 4910 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/14899c7c-f079-4e5a-b044-9cc72f305b83-archives-path\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:24.910508 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.910474 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" Nov 01 00:45:24.910508 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.910493 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b79g-s2nxb" event={"ID":"14899c7c-f079-4e5a-b044-9cc72f305b83","Type":"ContainerDied","Data":"0f45f8a5892220ec89f199e0c191071bc0edbec230364d2f0c855469c2584b93"} Nov 01 00:45:24.911099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.910560 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f45f8a5892220ec89f199e0c191071bc0edbec230364d2f0c855469c2584b93" Nov 01 00:45:24.912314 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.912287 4910 generic.go:356] "Generic (PLEG): container finished" podID="5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" containerID="573eaef2a18661115b7b55d9fb7c025c2f0b8fea9d48000fecdfe25c904fa154" exitCode=0 Nov 01 00:45:24.912447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:24.912383 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" event={"ID":"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef","Type":"ContainerDied","Data":"573eaef2a18661115b7b55d9fb7c025c2f0b8fea9d48000fecdfe25c904fa154"} Nov 01 00:45:25.019018 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.018985 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf"] Nov 01 00:45:25.019494 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019482 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:25.019541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019496 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:25.019541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019511 4910 cpu_manager.go:414] "containerMap: removing container" podUID="14899c7c-f079-4e5a-b044-9cc72f305b83" containerName="insights-gathering" Nov 01 00:45:25.019541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019516 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14899c7c-f079-4e5a-b044-9cc72f305b83" containerName="insights-gathering" Nov 01 00:45:25.019541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019525 4910 cpu_manager.go:414] "containerMap: removing container" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:25.019541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019530 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:25.019541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019537 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:25.019541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019542 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee4f1a2-9040-4505-8001-5f5a23a2d122" containerName="cleanup" Nov 01 00:45:25.019760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019549 4910 cpu_manager.go:414] "containerMap: removing container" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:25.019760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019553 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b5f06c-b6f6-4f13-b7ee-3f46090fbbef" containerName="cleanup" Nov 01 00:45:25.019760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019643 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda98f1c-115c-4625-8f42-8850439f3008" containerName="cleanup" Nov 01 00:45:25.019760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019654 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="15066416-112b-4dfd-8761-2cf5116b8021" containerName="cleanup" Nov 01 00:45:25.019760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019668 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="14899c7c-f079-4e5a-b044-9cc72f305b83" containerName="insights-gathering" Nov 01 00:45:25.019760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.019675 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95ed2b7-b0b6-4031-ab28-7d0cadddb621" containerName="cleanup" Nov 01 00:45:25.023041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.023024 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.087745 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.087716 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf"] Nov 01 00:45:25.106992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.106966 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.107144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.107009 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.107144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.107042 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6msqx\" (UniqueName: \"kubernetes.io/projected/027eb9e0-0d3f-41cb-b367-6581832f1f77-kube-api-access-6msqx\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.207973 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.207902 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.207973 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.207943 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.208133 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.208110 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6msqx\" (UniqueName: \"kubernetes.io/projected/027eb9e0-0d3f-41cb-b367-6581832f1f77-kube-api-access-6msqx\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.208536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.208516 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.210298 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.210278 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.269433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.269407 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6msqx\" (UniqueName: \"kubernetes.io/projected/027eb9e0-0d3f-41cb-b367-6581832f1f77-kube-api-access-6msqx\") pod \"maas-api-revoke-sub-820d88bb-68khf\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.335985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.335953 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:25.716914 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:25.716883 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027eb9e0_0d3f_41cb_b367_6581832f1f77.slice/crio-c45b0b542604da993ef26eecd82010b3e3a9bcb892e50c16e40b20fb298585e4 WatchSource:0}: Error finding container c45b0b542604da993ef26eecd82010b3e3a9bcb892e50c16e40b20fb298585e4: Status 404 returned error can't find the container with id c45b0b542604da993ef26eecd82010b3e3a9bcb892e50c16e40b20fb298585e4 Nov 01 00:45:25.717732 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.717705 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf"] Nov 01 00:45:25.920715 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:25.920679 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" event={"ID":"027eb9e0-0d3f-41cb-b367-6581832f1f77","Type":"ContainerStarted","Data":"c45b0b542604da993ef26eecd82010b3e3a9bcb892e50c16e40b20fb298585e4"} Nov 01 00:45:26.054958 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.054938 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:26.219541 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.219513 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-xks-ca\") pod \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " Nov 01 00:45:26.219705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.219560 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-openshift-ca\") pod \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " Nov 01 00:45:26.219705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.219610 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-kube-api-access-h2kzb\" (UniqueName: \"kubernetes.io/projected/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-kube-api-access-h2kzb\") pod \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\" (UID: \"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef\") " Nov 01 00:45:26.220001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.219976 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-openshift-ca" pod "5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" (UID: "5d5b18f4-2e9c-46f0-b439-d95b6e4161ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:26.220127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.220107 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:26.222052 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.222001 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-xks-ca" pod "5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" (UID: "5d5b18f4-2e9c-46f0-b439-d95b6e4161ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:26.222052 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.222007 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-kube-api-access-h2kzb" pod "5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" (UID: "5d5b18f4-2e9c-46f0-b439-d95b6e4161ef"). InnerVolumeSpecName "kube-api-access-h2kzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:26.320975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.320946 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:26.320975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.320970 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2kzb\" (UniqueName: \"kubernetes.io/projected/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef-kube-api-access-h2kzb\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:26.928718 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.928611 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" Nov 01 00:45:26.929210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.928613 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k" event={"ID":"5d5b18f4-2e9c-46f0-b439-d95b6e4161ef","Type":"ContainerDied","Data":"9b52bca198d6a93ff740bbf91cc557620ca1713da6bb8768bc922a3aa2fd55c7"} Nov 01 00:45:26.929210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.928778 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b52bca198d6a93ff740bbf91cc557620ca1713da6bb8768bc922a3aa2fd55c7" Nov 01 00:45:26.930537 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.930511 4910 generic.go:356] "Generic (PLEG): container finished" podID="027eb9e0-0d3f-41cb-b367-6581832f1f77" containerID="8b50ac040979cd653e8c615a86ad8b0f4510f794adecce7e378b70e4d330080b" exitCode=0 Nov 01 00:45:26.930696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:26.930547 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" event={"ID":"027eb9e0-0d3f-41cb-b367-6581832f1f77","Type":"ContainerDied","Data":"8b50ac040979cd653e8c615a86ad8b0f4510f794adecce7e378b70e4d330080b"} Nov 01 00:45:28.090573 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.090554 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:28.242179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.242072 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027eb9e0-0d3f-41cb-b367-6581832f1f77-kube-api-access-6msqx\" (UniqueName: \"kubernetes.io/projected/027eb9e0-0d3f-41cb-b367-6581832f1f77-kube-api-access-6msqx\") pod \"027eb9e0-0d3f-41cb-b367-6581832f1f77\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " Nov 01 00:45:28.242179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.242165 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-openshift-ca\") pod \"027eb9e0-0d3f-41cb-b367-6581832f1f77\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " Nov 01 00:45:28.242443 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.242426 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-openshift-ca" pod "027eb9e0-0d3f-41cb-b367-6581832f1f77" (UID: "027eb9e0-0d3f-41cb-b367-6581832f1f77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:28.242593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.242567 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-xks-ca\") pod \"027eb9e0-0d3f-41cb-b367-6581832f1f77\" (UID: \"027eb9e0-0d3f-41cb-b367-6581832f1f77\") " Nov 01 00:45:28.243069 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.243047 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:28.244764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.244739 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-xks-ca" pod "027eb9e0-0d3f-41cb-b367-6581832f1f77" (UID: "027eb9e0-0d3f-41cb-b367-6581832f1f77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:28.244928 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.244908 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027eb9e0-0d3f-41cb-b367-6581832f1f77-kube-api-access-6msqx" pod "027eb9e0-0d3f-41cb-b367-6581832f1f77" (UID: "027eb9e0-0d3f-41cb-b367-6581832f1f77"). InnerVolumeSpecName "kube-api-access-6msqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:28.343875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.343845 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027eb9e0-0d3f-41cb-b367-6581832f1f77-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:28.343875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.343871 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6msqx\" (UniqueName: \"kubernetes.io/projected/027eb9e0-0d3f-41cb-b367-6581832f1f77-kube-api-access-6msqx\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:28.487042 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.487010 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5"] Nov 01 00:45:28.487853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.487805 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" containerName="revoke-keys" Nov 01 00:45:28.487901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.487857 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" containerName="revoke-keys" Nov 01 00:45:28.487942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.487924 4910 cpu_manager.go:414] "containerMap: removing container" podUID="027eb9e0-0d3f-41cb-b367-6581832f1f77" containerName="revoke-keys" Nov 01 00:45:28.487942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.487932 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027eb9e0-0d3f-41cb-b367-6581832f1f77" containerName="revoke-keys" Nov 01 00:45:28.488081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.488060 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" containerName="revoke-keys" Nov 01 00:45:28.488185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.488087 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="027eb9e0-0d3f-41cb-b367-6581832f1f77" containerName="revoke-keys" Nov 01 00:45:28.491597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.491579 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.546460 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.546431 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5"] Nov 01 00:45:28.647273 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.647241 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.647440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.647284 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.647440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.647309 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rm25\" (UniqueName: \"kubernetes.io/projected/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-kube-api-access-2rm25\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.748186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.748159 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.748358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.748198 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.748358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.748220 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rm25\" (UniqueName: \"kubernetes.io/projected/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-kube-api-access-2rm25\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.749057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.748926 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.750602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.750579 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.810009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.809986 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rm25\" (UniqueName: \"kubernetes.io/projected/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-kube-api-access-2rm25\") pod \"maas-api-revoke-sub-10f71f39-rjdl5\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:28.945926 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.945895 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" Nov 01 00:45:28.946073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.945933 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf" event={"ID":"027eb9e0-0d3f-41cb-b367-6581832f1f77","Type":"ContainerDied","Data":"c45b0b542604da993ef26eecd82010b3e3a9bcb892e50c16e40b20fb298585e4"} Nov 01 00:45:28.946073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:28.946010 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45b0b542604da993ef26eecd82010b3e3a9bcb892e50c16e40b20fb298585e4" Nov 01 00:45:29.106073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:29.106005 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:29.482763 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:29.482735 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5"] Nov 01 00:45:29.483022 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:29.482998 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce8e33e_5c07_4c64_bd84_7bdc1b2f01ac.slice/crio-791a8b1268b2311da787c08ca85d23c7cb4c37cdf58cf06dfcde908b7f632a20 WatchSource:0}: Error finding container 791a8b1268b2311da787c08ca85d23c7cb4c37cdf58cf06dfcde908b7f632a20: Status 404 returned error can't find the container with id 791a8b1268b2311da787c08ca85d23c7cb4c37cdf58cf06dfcde908b7f632a20 Nov 01 00:45:29.955213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:29.955180 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" event={"ID":"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac","Type":"ContainerStarted","Data":"abadbca92e896c69c22557e8acfc37955653c71172373e3562d240bba0f063cd"} Nov 01 00:45:29.955374 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:29.955245 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" event={"ID":"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac","Type":"ContainerStarted","Data":"791a8b1268b2311da787c08ca85d23c7cb4c37cdf58cf06dfcde908b7f632a20"} Nov 01 00:45:29.957582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:29.957562 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7"] Nov 01 00:45:29.961572 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:29.961551 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.021798 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.021732 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7"] Nov 01 00:45:30.060890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.060848 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdwn\" (UniqueName: \"kubernetes.io/projected/8e68eb90-0395-4e19-8df6-28408b62b45d-kube-api-access-4vdwn\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.060890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.060893 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.061100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.061063 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.072180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.072139 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" podStartSLOduration=2.072127803 podStartE2EDuration="2.072127803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:30.070565124 +0000 UTC m=+1320.050147460" watchObservedRunningTime="2026-11-01 00:45:30.072127803 +0000 UTC m=+1320.051710138" Nov 01 00:45:30.161908 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.161858 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdwn\" (UniqueName: \"kubernetes.io/projected/8e68eb90-0395-4e19-8df6-28408b62b45d-kube-api-access-4vdwn\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.161908 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.161910 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.162441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.161955 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.162797 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.162776 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.164436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.164413 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.222561 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.222534 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdwn\" (UniqueName: \"kubernetes.io/projected/8e68eb90-0395-4e19-8df6-28408b62b45d-kube-api-access-4vdwn\") pod \"maas-api-revoke-sub-0b1f7581-ztjs7\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.276170 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.276085 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:30.658578 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.658549 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7"] Nov 01 00:45:30.658904 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:30.658878 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e68eb90_0395_4e19_8df6_28408b62b45d.slice/crio-9ce4d4ca5147be0ad5a5d6d1324e3933e87a14e64cd83032824c84b6d12cf018 WatchSource:0}: Error finding container 9ce4d4ca5147be0ad5a5d6d1324e3933e87a14e64cd83032824c84b6d12cf018: Status 404 returned error can't find the container with id 9ce4d4ca5147be0ad5a5d6d1324e3933e87a14e64cd83032824c84b6d12cf018 Nov 01 00:45:30.963785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.963750 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" event={"ID":"8e68eb90-0395-4e19-8df6-28408b62b45d","Type":"ContainerStarted","Data":"9ce4d4ca5147be0ad5a5d6d1324e3933e87a14e64cd83032824c84b6d12cf018"} Nov 01 00:45:30.965710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.965682 4910 generic.go:356] "Generic (PLEG): container finished" podID="4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" containerID="abadbca92e896c69c22557e8acfc37955653c71172373e3562d240bba0f063cd" exitCode=0 Nov 01 00:45:30.965851 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:30.965833 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" event={"ID":"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac","Type":"ContainerDied","Data":"abadbca92e896c69c22557e8acfc37955653c71172373e3562d240bba0f063cd"} Nov 01 00:45:31.984542 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:31.984451 4910 generic.go:356] "Generic (PLEG): container finished" podID="8e68eb90-0395-4e19-8df6-28408b62b45d" containerID="1b19a2d0aaf0c4d3a4638feea09fe010a3c7f2861d4de04530226f9278559cc4" exitCode=0 Nov 01 00:45:31.984542 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:31.984523 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" event={"ID":"8e68eb90-0395-4e19-8df6-28408b62b45d","Type":"ContainerDied","Data":"1b19a2d0aaf0c4d3a4638feea09fe010a3c7f2861d4de04530226f9278559cc4"} Nov 01 00:45:32.148030 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.148003 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:32.286604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.286564 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-kube-api-access-2rm25\" (UniqueName: \"kubernetes.io/projected/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-kube-api-access-2rm25\") pod \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " Nov 01 00:45:32.286817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.286684 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-xks-ca\") pod \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " Nov 01 00:45:32.286817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.286787 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-openshift-ca\") pod \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\" (UID: \"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac\") " Nov 01 00:45:32.287196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.287166 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-openshift-ca" pod "4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" (UID: "4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:32.288957 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.288933 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-kube-api-access-2rm25" pod "4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" (UID: "4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac"). InnerVolumeSpecName "kube-api-access-2rm25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:32.289592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.289558 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-xks-ca" pod "4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" (UID: "4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:32.388370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.388340 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rm25\" (UniqueName: \"kubernetes.io/projected/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-kube-api-access-2rm25\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:32.388370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.388364 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:32.388370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.388374 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:32.767416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.767384 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc"] Nov 01 00:45:32.767971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.767955 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" containerName="revoke-keys" Nov 01 00:45:32.768021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.767973 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" containerName="revoke-keys" Nov 01 00:45:32.768061 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.768047 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" containerName="revoke-keys" Nov 01 00:45:32.771517 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.771498 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.829279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.829245 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc"] Nov 01 00:45:32.893907 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.893872 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.894074 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.894013 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.894074 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.894044 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c65k\" (UniqueName: \"kubernetes.io/projected/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-kube-api-access-2c65k\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.992759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.992724 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" Nov 01 00:45:32.993247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.992760 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5" event={"ID":"4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac","Type":"ContainerDied","Data":"791a8b1268b2311da787c08ca85d23c7cb4c37cdf58cf06dfcde908b7f632a20"} Nov 01 00:45:32.993247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.992802 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="791a8b1268b2311da787c08ca85d23c7cb4c37cdf58cf06dfcde908b7f632a20" Nov 01 00:45:32.995122 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.995088 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.995346 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.995323 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.995422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.995377 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c65k\" (UniqueName: \"kubernetes.io/projected/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-kube-api-access-2c65k\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.996111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.996087 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:32.997831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:32.997811 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:33.061785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.061702 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c65k\" (UniqueName: \"kubernetes.io/projected/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-kube-api-access-2c65k\") pod \"maas-api-revoke-sub-bf1fcf80-r5blc\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:33.084803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.084776 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:33.140731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.140708 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:33.298150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.298113 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e68eb90-0395-4e19-8df6-28408b62b45d-kube-api-access-4vdwn\" (UniqueName: \"kubernetes.io/projected/8e68eb90-0395-4e19-8df6-28408b62b45d-kube-api-access-4vdwn\") pod \"8e68eb90-0395-4e19-8df6-28408b62b45d\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " Nov 01 00:45:33.298331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.298302 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-openshift-ca\") pod \"8e68eb90-0395-4e19-8df6-28408b62b45d\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " Nov 01 00:45:33.298399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.298340 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-xks-ca\") pod \"8e68eb90-0395-4e19-8df6-28408b62b45d\" (UID: \"8e68eb90-0395-4e19-8df6-28408b62b45d\") " Nov 01 00:45:33.298764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.298732 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-openshift-ca" pod "8e68eb90-0395-4e19-8df6-28408b62b45d" (UID: "8e68eb90-0395-4e19-8df6-28408b62b45d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:33.300905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.300847 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-xks-ca" pod "8e68eb90-0395-4e19-8df6-28408b62b45d" (UID: "8e68eb90-0395-4e19-8df6-28408b62b45d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:33.301309 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.301289 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e68eb90-0395-4e19-8df6-28408b62b45d-kube-api-access-4vdwn" pod "8e68eb90-0395-4e19-8df6-28408b62b45d" (UID: "8e68eb90-0395-4e19-8df6-28408b62b45d"). InnerVolumeSpecName "kube-api-access-4vdwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:33.400045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.400014 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:33.400045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.400048 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e68eb90-0395-4e19-8df6-28408b62b45d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:33.400206 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.400062 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vdwn\" (UniqueName: \"kubernetes.io/projected/8e68eb90-0395-4e19-8df6-28408b62b45d-kube-api-access-4vdwn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:33.470562 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:33.470521 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01c4e4e_9c3f_4b2c_9f2d_5f74782fd718.slice/crio-04bfd6002b45ec4953b49d56c64be30d71c23cd935fdaf0203ae56fbb518ebb0 WatchSource:0}: Error finding container 04bfd6002b45ec4953b49d56c64be30d71c23cd935fdaf0203ae56fbb518ebb0: Status 404 returned error can't find the container with id 04bfd6002b45ec4953b49d56c64be30d71c23cd935fdaf0203ae56fbb518ebb0 Nov 01 00:45:33.472293 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:33.472220 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc"] Nov 01 00:45:34.000198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:34.000172 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" Nov 01 00:45:34.000604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:34.000206 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7" event={"ID":"8e68eb90-0395-4e19-8df6-28408b62b45d","Type":"ContainerDied","Data":"9ce4d4ca5147be0ad5a5d6d1324e3933e87a14e64cd83032824c84b6d12cf018"} Nov 01 00:45:34.000604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:34.000258 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ce4d4ca5147be0ad5a5d6d1324e3933e87a14e64cd83032824c84b6d12cf018" Nov 01 00:45:34.001829 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:34.001796 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" event={"ID":"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718","Type":"ContainerStarted","Data":"2290f209239740745607e753d50235bd0ffa98a2be3130b43117e1682d39761e"} Nov 01 00:45:34.001916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:34.001849 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" event={"ID":"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718","Type":"ContainerStarted","Data":"04bfd6002b45ec4953b49d56c64be30d71c23cd935fdaf0203ae56fbb518ebb0"} Nov 01 00:45:34.120299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:34.120251 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" podStartSLOduration=2.120235918 podStartE2EDuration="2.120235918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:34.117010315 +0000 UTC m=+1324.096592652" watchObservedRunningTime="2026-11-01 00:45:34.120235918 +0000 UTC m=+1324.099818253" Nov 01 00:45:35.018497 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.018465 4910 generic.go:356] "Generic (PLEG): container finished" podID="d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" containerID="2290f209239740745607e753d50235bd0ffa98a2be3130b43117e1682d39761e" exitCode=0 Nov 01 00:45:35.019010 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.018555 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" event={"ID":"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718","Type":"ContainerDied","Data":"2290f209239740745607e753d50235bd0ffa98a2be3130b43117e1682d39761e"} Nov 01 00:45:35.779898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.779868 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg"] Nov 01 00:45:35.780411 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.780398 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8e68eb90-0395-4e19-8df6-28408b62b45d" containerName="revoke-keys" Nov 01 00:45:35.780452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.780413 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e68eb90-0395-4e19-8df6-28408b62b45d" containerName="revoke-keys" Nov 01 00:45:35.780500 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.780491 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e68eb90-0395-4e19-8df6-28408b62b45d" containerName="revoke-keys" Nov 01 00:45:35.783904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.783885 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:35.841689 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.841661 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg"] Nov 01 00:45:35.926088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.926057 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:35.926240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.926102 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9wx\" (UniqueName: \"kubernetes.io/projected/e20f6454-44fb-46ea-aec7-63a349190c59-kube-api-access-4j9wx\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:35.926240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:35.926135 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.027388 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.027366 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9wx\" (UniqueName: \"kubernetes.io/projected/e20f6454-44fb-46ea-aec7-63a349190c59-kube-api-access-4j9wx\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.027679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.027403 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.027679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.027573 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.028414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.028364 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.030349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.030295 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.093297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.093265 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9wx\" (UniqueName: \"kubernetes.io/projected/e20f6454-44fb-46ea-aec7-63a349190c59-kube-api-access-4j9wx\") pod \"maas-api-revoke-sub-caed0494-9xfsg\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.097970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.097944 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:36.206037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.206006 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:36.330801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.330692 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-kube-api-access-2c65k\" (UniqueName: \"kubernetes.io/projected/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-kube-api-access-2c65k\") pod \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " Nov 01 00:45:36.330995 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.330852 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-xks-ca\") pod \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " Nov 01 00:45:36.330995 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.330900 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-openshift-ca\") pod \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\" (UID: \"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718\") " Nov 01 00:45:36.331589 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.331547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-openshift-ca" pod "d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" (UID: "d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:36.333347 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.333312 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-kube-api-access-2c65k" pod "d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" (UID: "d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718"). InnerVolumeSpecName "kube-api-access-2c65k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:36.333455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.333435 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-xks-ca" pod "d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" (UID: "d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:36.432544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.432514 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:36.432544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.432539 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:36.432544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.432550 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c65k\" (UniqueName: \"kubernetes.io/projected/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718-kube-api-access-2c65k\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:36.486689 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:36.486645 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20f6454_44fb_46ea_aec7_63a349190c59.slice/crio-2db3c2789d83f61a896b5bc30c86a0ce45542450ed2c0323d972b810719fb695 WatchSource:0}: Error finding container 2db3c2789d83f61a896b5bc30c86a0ce45542450ed2c0323d972b810719fb695: Status 404 returned error can't find the container with id 2db3c2789d83f61a896b5bc30c86a0ce45542450ed2c0323d972b810719fb695 Nov 01 00:45:36.487540 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:36.487507 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg"] Nov 01 00:45:37.033781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:37.033747 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" event={"ID":"e20f6454-44fb-46ea-aec7-63a349190c59","Type":"ContainerStarted","Data":"653d62081f55017950a834f0c8e06ad34526c6a4f5fa20e725127e809964a7ea"} Nov 01 00:45:37.034220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:37.033805 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" event={"ID":"e20f6454-44fb-46ea-aec7-63a349190c59","Type":"ContainerStarted","Data":"2db3c2789d83f61a896b5bc30c86a0ce45542450ed2c0323d972b810719fb695"} Nov 01 00:45:37.035220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:37.035194 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" event={"ID":"d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718","Type":"ContainerDied","Data":"04bfd6002b45ec4953b49d56c64be30d71c23cd935fdaf0203ae56fbb518ebb0"} Nov 01 00:45:37.035282 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:37.035222 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc" Nov 01 00:45:37.035282 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:37.035244 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04bfd6002b45ec4953b49d56c64be30d71c23cd935fdaf0203ae56fbb518ebb0" Nov 01 00:45:37.153231 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:37.153171 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" podStartSLOduration=2.153158021 podStartE2EDuration="2.153158021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:37.149270109 +0000 UTC m=+1327.128852445" watchObservedRunningTime="2026-11-01 00:45:37.153158021 +0000 UTC m=+1327.132740355" Nov 01 00:45:38.044258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:38.044220 4910 generic.go:356] "Generic (PLEG): container finished" podID="e20f6454-44fb-46ea-aec7-63a349190c59" containerID="653d62081f55017950a834f0c8e06ad34526c6a4f5fa20e725127e809964a7ea" exitCode=0 Nov 01 00:45:38.044654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:38.044275 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" event={"ID":"e20f6454-44fb-46ea-aec7-63a349190c59","Type":"ContainerDied","Data":"653d62081f55017950a834f0c8e06ad34526c6a4f5fa20e725127e809964a7ea"} Nov 01 00:45:38.957960 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:38.957924 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6"] Nov 01 00:45:38.958811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:38.958722 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" containerName="revoke-keys" Nov 01 00:45:38.958811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:38.958744 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" containerName="revoke-keys" Nov 01 00:45:38.958982 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:38.958924 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" containerName="revoke-keys" Nov 01 00:45:38.985957 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:38.985925 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7pjw\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-kube-api-access-f7pjw\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059165 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059283 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059320 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059431 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6096227-de28-416a-a1cf-909227cd52d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059465 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6096227-de28-416a-a1cf-909227cd52d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059557 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059612 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.059684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.059663 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.103195 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.102443 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6"] Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160424 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6096227-de28-416a-a1cf-909227cd52d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160474 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160568 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160615 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7pjw\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-kube-api-access-f7pjw\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160669 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160709 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160740 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.160829 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6096227-de28-416a-a1cf-909227cd52d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.161982 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.163042 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.163264 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.163770 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.164227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.163955 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6096227-de28-416a-a1cf-909227cd52d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.167665 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.166550 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6096227-de28-416a-a1cf-909227cd52d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.168573 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.168553 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.227826 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.227715 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7pjw\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-kube-api-access-f7pjw\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.228225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.228202 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cn8t6\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.272897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.272872 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:39.290409 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.290375 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t"] Nov 01 00:45:39.290956 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.290941 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e20f6454-44fb-46ea-aec7-63a349190c59" containerName="revoke-keys" Nov 01 00:45:39.290956 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.290957 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20f6454-44fb-46ea-aec7-63a349190c59" containerName="revoke-keys" Nov 01 00:45:39.291109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.291036 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20f6454-44fb-46ea-aec7-63a349190c59" containerName="revoke-keys" Nov 01 00:45:39.294307 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.294292 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.332064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.332021 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:39.358082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.356884 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t"] Nov 01 00:45:39.362721 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.362685 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20f6454-44fb-46ea-aec7-63a349190c59-kube-api-access-4j9wx\" (UniqueName: \"kubernetes.io/projected/e20f6454-44fb-46ea-aec7-63a349190c59-kube-api-access-4j9wx\") pod \"e20f6454-44fb-46ea-aec7-63a349190c59\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " Nov 01 00:45:39.362866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.362740 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-openshift-ca\") pod \"e20f6454-44fb-46ea-aec7-63a349190c59\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " Nov 01 00:45:39.362866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.362776 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-xks-ca\") pod \"e20f6454-44fb-46ea-aec7-63a349190c59\" (UID: \"e20f6454-44fb-46ea-aec7-63a349190c59\") " Nov 01 00:45:39.366664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.363082 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.366664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.363139 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlxz\" (UniqueName: \"kubernetes.io/projected/48a24ae4-683d-4664-ade8-24a0cdf3832e-kube-api-access-7xlxz\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.366664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.363328 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-openshift-ca" pod "e20f6454-44fb-46ea-aec7-63a349190c59" (UID: "e20f6454-44fb-46ea-aec7-63a349190c59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:39.366664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.363520 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.366664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.363926 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:39.366664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.365426 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20f6454-44fb-46ea-aec7-63a349190c59-kube-api-access-4j9wx" pod "e20f6454-44fb-46ea-aec7-63a349190c59" (UID: "e20f6454-44fb-46ea-aec7-63a349190c59"). InnerVolumeSpecName "kube-api-access-4j9wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:39.371007 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.370983 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-xks-ca" pod "e20f6454-44fb-46ea-aec7-63a349190c59" (UID: "e20f6454-44fb-46ea-aec7-63a349190c59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:39.464656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.464588 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.464816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.464733 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.464816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.464764 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlxz\" (UniqueName: \"kubernetes.io/projected/48a24ae4-683d-4664-ade8-24a0cdf3832e-kube-api-access-7xlxz\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.464991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.464848 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9wx\" (UniqueName: \"kubernetes.io/projected/e20f6454-44fb-46ea-aec7-63a349190c59-kube-api-access-4j9wx\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:39.464991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.464859 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20f6454-44fb-46ea-aec7-63a349190c59-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:39.465690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.465666 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.468343 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.468316 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.526089 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.526060 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlxz\" (UniqueName: \"kubernetes.io/projected/48a24ae4-683d-4664-ade8-24a0cdf3832e-kube-api-access-7xlxz\") pod \"maas-api-revoke-sub-fe3b9220-mq98t\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.613591 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.613558 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:39.813304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.813240 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6"] Nov 01 00:45:39.815949 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.815910 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:39.816064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.816020 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:39.816127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:39.816068 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:40.046656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.046183 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t"] Nov 01 00:45:40.046978 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:40.046905 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a24ae4_683d_4664_ade8_24a0cdf3832e.slice/crio-90b47e1562595496c73b36c0ac786c6e449820f5aafa5fb2015bf46810c34096 WatchSource:0}: Error finding container 90b47e1562595496c73b36c0ac786c6e449820f5aafa5fb2015bf46810c34096: Status 404 returned error can't find the container with id 90b47e1562595496c73b36c0ac786c6e449820f5aafa5fb2015bf46810c34096 Nov 01 00:45:40.099389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.098432 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" event={"ID":"c6096227-de28-416a-a1cf-909227cd52d9","Type":"ContainerStarted","Data":"6abcb4bc28096cac36e2b2d7e1e99f6bbeaac66752ea034322a99764f960f524"} Nov 01 00:45:40.099389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.098497 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" event={"ID":"c6096227-de28-416a-a1cf-909227cd52d9","Type":"ContainerStarted","Data":"d1d66d33ed1cfe2a3de56b728ccf49e952598531f08508ffb69f680f9441a76c"} Nov 01 00:45:40.101839 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.101812 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" event={"ID":"e20f6454-44fb-46ea-aec7-63a349190c59","Type":"ContainerDied","Data":"2db3c2789d83f61a896b5bc30c86a0ce45542450ed2c0323d972b810719fb695"} Nov 01 00:45:40.104673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.101981 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db3c2789d83f61a896b5bc30c86a0ce45542450ed2c0323d972b810719fb695" Nov 01 00:45:40.104673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.102045 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg" Nov 01 00:45:40.124792 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.124656 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" event={"ID":"48a24ae4-683d-4664-ade8-24a0cdf3832e","Type":"ContainerStarted","Data":"90b47e1562595496c73b36c0ac786c6e449820f5aafa5fb2015bf46810c34096"} Nov 01 00:45:40.240917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.240858 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podStartSLOduration=2.2408391229999998 podStartE2EDuration="2.240839123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:40.239178522 +0000 UTC m=+1330.218760856" watchObservedRunningTime="2026-11-01 00:45:40.240839123 +0000 UTC m=+1330.220421459" Nov 01 00:45:40.333101 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.333002 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:40.334252 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.334227 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:40.334367 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:40.334275 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:41.132945 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:41.132911 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" event={"ID":"48a24ae4-683d-4664-ade8-24a0cdf3832e","Type":"ContainerStarted","Data":"2580574e00cbffeb0add5084305073a6ce705ef51612f55b459544967280131c"} Nov 01 00:45:41.280400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:41.280327 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" podStartSLOduration=2.280307139 podStartE2EDuration="2.280307139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:41.248716882 +0000 UTC m=+1331.228299218" watchObservedRunningTime="2026-11-01 00:45:41.280307139 +0000 UTC m=+1331.259889477" Nov 01 00:45:41.333062 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:41.333022 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:41.333244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:41.333075 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:42.141353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:42.141315 4910 generic.go:356] "Generic (PLEG): container finished" podID="48a24ae4-683d-4664-ade8-24a0cdf3832e" containerID="2580574e00cbffeb0add5084305073a6ce705ef51612f55b459544967280131c" exitCode=0 Nov 01 00:45:42.141810 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:42.141409 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" event={"ID":"48a24ae4-683d-4664-ade8-24a0cdf3832e","Type":"ContainerDied","Data":"2580574e00cbffeb0add5084305073a6ce705ef51612f55b459544967280131c"} Nov 01 00:45:42.333099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:42.333063 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:42.333262 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:42.333131 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:43.109276 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:45:43.109242 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:45:43.332975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.332943 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:43.333381 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.332993 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:43.389690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.389577 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:43.474648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.474023 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a24ae4-683d-4664-ade8-24a0cdf3832e-kube-api-access-7xlxz\" (UniqueName: \"kubernetes.io/projected/48a24ae4-683d-4664-ade8-24a0cdf3832e-kube-api-access-7xlxz\") pod \"48a24ae4-683d-4664-ade8-24a0cdf3832e\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " Nov 01 00:45:43.474648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.474152 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-xks-ca\") pod \"48a24ae4-683d-4664-ade8-24a0cdf3832e\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " Nov 01 00:45:43.474648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.474194 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-openshift-ca\") pod \"48a24ae4-683d-4664-ade8-24a0cdf3832e\" (UID: \"48a24ae4-683d-4664-ade8-24a0cdf3832e\") " Nov 01 00:45:43.474967 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.474902 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-openshift-ca" pod "48a24ae4-683d-4664-ade8-24a0cdf3832e" (UID: "48a24ae4-683d-4664-ade8-24a0cdf3832e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:43.483900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.483839 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-xks-ca" pod "48a24ae4-683d-4664-ade8-24a0cdf3832e" (UID: "48a24ae4-683d-4664-ade8-24a0cdf3832e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:43.484781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.484753 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a24ae4-683d-4664-ade8-24a0cdf3832e-kube-api-access-7xlxz" pod "48a24ae4-683d-4664-ade8-24a0cdf3832e" (UID: "48a24ae4-683d-4664-ade8-24a0cdf3832e"). InnerVolumeSpecName "kube-api-access-7xlxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:43.586482 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.586445 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:43.586721 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.586704 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a24ae4-683d-4664-ade8-24a0cdf3832e-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:43.586809 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:43.586730 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xlxz\" (UniqueName: \"kubernetes.io/projected/48a24ae4-683d-4664-ade8-24a0cdf3832e-kube-api-access-7xlxz\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:44.181568 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:44.181533 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" event={"ID":"48a24ae4-683d-4664-ade8-24a0cdf3832e","Type":"ContainerDied","Data":"90b47e1562595496c73b36c0ac786c6e449820f5aafa5fb2015bf46810c34096"} Nov 01 00:45:44.181901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:44.181881 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b47e1562595496c73b36c0ac786c6e449820f5aafa5fb2015bf46810c34096" Nov 01 00:45:44.182081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:44.182068 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t" Nov 01 00:45:44.333496 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:44.333459 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:44.333947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:44.333523 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:45.333047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.333015 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:45.333240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.333070 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:45.620032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.619959 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq"] Nov 01 00:45:45.620489 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.620476 4910 cpu_manager.go:414] "containerMap: removing container" podUID="48a24ae4-683d-4664-ade8-24a0cdf3832e" containerName="revoke-keys" Nov 01 00:45:45.620531 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.620490 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a24ae4-683d-4664-ade8-24a0cdf3832e" containerName="revoke-keys" Nov 01 00:45:45.620597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.620587 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a24ae4-683d-4664-ade8-24a0cdf3832e" containerName="revoke-keys" Nov 01 00:45:45.624329 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.624314 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.692635 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.692562 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq"] Nov 01 00:45:45.709822 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.709790 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.709983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.709899 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzp7\" (UniqueName: \"kubernetes.io/projected/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-kube-api-access-vbzp7\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.710029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.710011 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.727085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.727025 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:45:45.727085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.727062 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:45:45.811145 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.811113 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.811376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.811236 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.811376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.811291 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzp7\" (UniqueName: \"kubernetes.io/projected/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-kube-api-access-vbzp7\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.812071 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.812047 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.814471 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.814436 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.880451 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.880301 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzp7\" (UniqueName: \"kubernetes.io/projected/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-kube-api-access-vbzp7\") pod \"maas-api-revoke-sub-382b03ad-fwchq\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:45.938013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:45.937982 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:46.333601 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:46.333569 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:46.333838 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:46.333657 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:46.347275 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:46.347248 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq"] Nov 01 00:45:46.348166 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:46.348142 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f05ff2e_a2a2_4d29_8910_c1c1a7bba0c6.slice/crio-f607408148db961a468c6435ad1ea312fab4fd3cd121543aac8c6050923e0c06 WatchSource:0}: Error finding container f607408148db961a468c6435ad1ea312fab4fd3cd121543aac8c6050923e0c06: Status 404 returned error can't find the container with id f607408148db961a468c6435ad1ea312fab4fd3cd121543aac8c6050923e0c06 Nov 01 00:45:47.214655 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:47.214591 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" event={"ID":"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6","Type":"ContainerStarted","Data":"58d965542eb45d2a006aabfa91004e48c24f111f122273ba13aedae4bcc808d2"} Nov 01 00:45:47.215052 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:47.214711 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" event={"ID":"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6","Type":"ContainerStarted","Data":"f607408148db961a468c6435ad1ea312fab4fd3cd121543aac8c6050923e0c06"} Nov 01 00:45:47.333105 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:47.333059 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" podStartSLOduration=2.333045172 podStartE2EDuration="2.333045172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:47.330111245 +0000 UTC m=+1337.309693591" watchObservedRunningTime="2026-11-01 00:45:47.333045172 +0000 UTC m=+1337.312627507" Nov 01 00:45:47.333526 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:47.333498 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cn8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 01 00:45:47.333672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:47.333556 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 01 00:45:48.180229 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.180190 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6"] Nov 01 00:45:48.185607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.185588 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.226105 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.226072 4910 generic.go:356] "Generic (PLEG): container finished" podID="1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" containerID="58d965542eb45d2a006aabfa91004e48c24f111f122273ba13aedae4bcc808d2" exitCode=0 Nov 01 00:45:48.226486 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.226170 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" event={"ID":"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6","Type":"ContainerDied","Data":"58d965542eb45d2a006aabfa91004e48c24f111f122273ba13aedae4bcc808d2"} Nov 01 00:45:48.241816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.241789 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6"] Nov 01 00:45:48.337986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.337946 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.338152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.338003 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbt22\" (UniqueName: \"kubernetes.io/projected/2b13b468-aca8-4367-bdc3-f109f8942aea-kube-api-access-cbt22\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.338243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.338220 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.439679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.439583 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.439679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.439646 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbt22\" (UniqueName: \"kubernetes.io/projected/2b13b468-aca8-4367-bdc3-f109f8942aea-kube-api-access-cbt22\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.439904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.439764 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.440474 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.440445 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.442370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.442347 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.485033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.485004 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:48.485493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.485469 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:48.486891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.486873 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:45:48.501243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.501224 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbt22\" (UniqueName: \"kubernetes.io/projected/2b13b468-aca8-4367-bdc3-f109f8942aea-kube-api-access-cbt22\") pod \"maas-api-revoke-sub-76554d1a-gl9v6\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:48.800820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:48.800787 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:49.182571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.182546 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6"] Nov 01 00:45:49.182891 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:49.182869 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b13b468_aca8_4367_bdc3_f109f8942aea.slice/crio-568345b1fc67b24a91bea03d5e65d48d325f7d27c7e5912442d443a600d48bc3 WatchSource:0}: Error finding container 568345b1fc67b24a91bea03d5e65d48d325f7d27c7e5912442d443a600d48bc3: Status 404 returned error can't find the container with id 568345b1fc67b24a91bea03d5e65d48d325f7d27c7e5912442d443a600d48bc3 Nov 01 00:45:49.235301 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.235271 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" event={"ID":"2b13b468-aca8-4367-bdc3-f109f8942aea","Type":"ContainerStarted","Data":"568345b1fc67b24a91bea03d5e65d48d325f7d27c7e5912442d443a600d48bc3"} Nov 01 00:45:49.369034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.369011 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:49.451067 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.451040 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-xks-ca\") pod \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " Nov 01 00:45:49.451129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.451109 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-kube-api-access-vbzp7\" (UniqueName: \"kubernetes.io/projected/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-kube-api-access-vbzp7\") pod \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " Nov 01 00:45:49.451399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.451266 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-openshift-ca\") pod \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\" (UID: \"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6\") " Nov 01 00:45:49.451909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.451881 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-openshift-ca" pod "1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" (UID: "1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:49.453318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.453289 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-xks-ca" pod "1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" (UID: "1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:49.453697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.453649 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-kube-api-access-vbzp7" pod "1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" (UID: "1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6"). InnerVolumeSpecName "kube-api-access-vbzp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:49.552685 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.552615 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:49.552685 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.552678 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbzp7\" (UniqueName: \"kubernetes.io/projected/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-kube-api-access-vbzp7\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:49.552860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:49.552694 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:50.243970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:50.243940 4910 generic.go:356] "Generic (PLEG): container finished" podID="2b13b468-aca8-4367-bdc3-f109f8942aea" containerID="d9a32ab3cad582dfa68da5a1bdc34f116d8fe96b8e4abc90e57362ea2c3f5f74" exitCode=0 Nov 01 00:45:50.244471 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:50.243982 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" event={"ID":"2b13b468-aca8-4367-bdc3-f109f8942aea","Type":"ContainerDied","Data":"d9a32ab3cad582dfa68da5a1bdc34f116d8fe96b8e4abc90e57362ea2c3f5f74"} Nov 01 00:45:50.245700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:50.245669 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" event={"ID":"1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6","Type":"ContainerDied","Data":"f607408148db961a468c6435ad1ea312fab4fd3cd121543aac8c6050923e0c06"} Nov 01 00:45:50.245700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:50.245691 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq" Nov 01 00:45:50.245876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:50.245734 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f607408148db961a468c6435ad1ea312fab4fd3cd121543aac8c6050923e0c06" Nov 01 00:45:51.401081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.401060 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:51.574567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.574490 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-xks-ca\") pod \"2b13b468-aca8-4367-bdc3-f109f8942aea\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " Nov 01 00:45:51.574567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.574535 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-openshift-ca\") pod \"2b13b468-aca8-4367-bdc3-f109f8942aea\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " Nov 01 00:45:51.574823 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.574593 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b13b468-aca8-4367-bdc3-f109f8942aea-kube-api-access-cbt22\" (UniqueName: \"kubernetes.io/projected/2b13b468-aca8-4367-bdc3-f109f8942aea-kube-api-access-cbt22\") pod \"2b13b468-aca8-4367-bdc3-f109f8942aea\" (UID: \"2b13b468-aca8-4367-bdc3-f109f8942aea\") " Nov 01 00:45:51.574945 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.574921 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-openshift-ca" pod "2b13b468-aca8-4367-bdc3-f109f8942aea" (UID: "2b13b468-aca8-4367-bdc3-f109f8942aea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:45:51.575107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.575087 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:51.576902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.576879 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-xks-ca" pod "2b13b468-aca8-4367-bdc3-f109f8942aea" (UID: "2b13b468-aca8-4367-bdc3-f109f8942aea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:45:51.577008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.576902 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b13b468-aca8-4367-bdc3-f109f8942aea-kube-api-access-cbt22" pod "2b13b468-aca8-4367-bdc3-f109f8942aea" (UID: "2b13b468-aca8-4367-bdc3-f109f8942aea"). InnerVolumeSpecName "kube-api-access-cbt22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:45:51.676617 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.676582 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b13b468-aca8-4367-bdc3-f109f8942aea-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:51.676617 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:51.676612 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbt22\" (UniqueName: \"kubernetes.io/projected/2b13b468-aca8-4367-bdc3-f109f8942aea-kube-api-access-cbt22\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:45:52.261323 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:52.261300 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" Nov 01 00:45:52.261479 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:52.261335 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6" event={"ID":"2b13b468-aca8-4367-bdc3-f109f8942aea","Type":"ContainerDied","Data":"568345b1fc67b24a91bea03d5e65d48d325f7d27c7e5912442d443a600d48bc3"} Nov 01 00:45:52.261479 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:52.261381 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="568345b1fc67b24a91bea03d5e65d48d325f7d27c7e5912442d443a600d48bc3" Nov 01 00:45:54.075542 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.075512 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr"] Nov 01 00:45:54.076022 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.076008 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" containerName="revoke-keys" Nov 01 00:45:54.076066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.076023 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" containerName="revoke-keys" Nov 01 00:45:54.076066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.076046 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2b13b468-aca8-4367-bdc3-f109f8942aea" containerName="revoke-keys" Nov 01 00:45:54.076066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.076050 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b13b468-aca8-4367-bdc3-f109f8942aea" containerName="revoke-keys" Nov 01 00:45:54.076163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.076146 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b13b468-aca8-4367-bdc3-f109f8942aea" containerName="revoke-keys" Nov 01 00:45:54.076163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.076153 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" containerName="revoke-keys" Nov 01 00:45:54.079729 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.079713 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201183 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201154 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzz5\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-kube-api-access-kgzz5\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201321 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201276 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201321 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201298 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201419 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201376 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201434 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201460 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201477 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201516 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.201597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.201585 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.283015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.282984 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr"] Nov 01 00:45:54.302550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302508 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.302753 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302554 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.302753 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302581 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.302753 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302601 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.302753 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302654 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.302753 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302699 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302756 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzz5\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-kube-api-access-kgzz5\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302870 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302895 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.302987 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.303583 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.303669 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303933 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.303742 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.303933 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.303766 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.305424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.305401 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.305793 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.305770 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.363785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.363726 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.363903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.363801 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzz5\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-kube-api-access-kgzz5\") pod \"maas-default-gateway-openshift-default-67bf986f99-fvzwr\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.392493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.392471 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:54.833255 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:45:54.833227 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f7c43a_37d1_446f_b8c6_445a71575fe8.slice/crio-f7c197dddf02a7eb657718ccf3167b0424116b9da0d78866a0902f659194c472 WatchSource:0}: Error finding container f7c197dddf02a7eb657718ccf3167b0424116b9da0d78866a0902f659194c472: Status 404 returned error can't find the container with id f7c197dddf02a7eb657718ccf3167b0424116b9da0d78866a0902f659194c472 Nov 01 00:45:54.833994 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.833953 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr"] Nov 01 00:45:54.835744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.835709 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:54.835844 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.835789 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:54.835844 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:54.835838 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:45:55.285707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:55.285661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" event={"ID":"c2f7c43a-37d1-446f-b8c6-445a71575fe8","Type":"ContainerStarted","Data":"481cbd4226e508de0b7df0109be53d7853fad31101c25efa6b6f23afdbb6c7ac"} Nov 01 00:45:55.286178 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:55.285734 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" event={"ID":"c2f7c43a-37d1-446f-b8c6-445a71575fe8","Type":"ContainerStarted","Data":"f7c197dddf02a7eb657718ccf3167b0424116b9da0d78866a0902f659194c472"} Nov 01 00:45:55.393780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:55.393746 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:45:55.395257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:55.395236 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:45:55.395313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:55.395283 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:45:55.412904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:55.412849 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podStartSLOduration=1.412837327 podStartE2EDuration="1.412837327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:45:55.408539485 +0000 UTC m=+1345.388121823" watchObservedRunningTime="2026-11-01 00:45:55.412837327 +0000 UTC m=+1345.392419731" Nov 01 00:45:56.393077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:56.393042 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:45:56.393591 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:56.393105 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:45:57.393702 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:57.393673 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:45:57.394299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:57.394273 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:45:58.393220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:58.393186 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:45:58.393440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:58.393252 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:45:59.393645 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:59.393363 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:45:59.393645 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:45:59.393420 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:46:00.413650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:00.411607 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:46:00.413650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:00.411736 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:46:01.393111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.393074 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:46:01.393326 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.393161 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:46:01.793352 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.793322 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx"] Nov 01 00:46:01.797937 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.797909 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.843202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.843158 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz"] Nov 01 00:46:01.847548 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.847524 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.851318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.851294 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-948e22-openshift-default-dockercfg-2wgzk\"" Nov 01 00:46:01.899470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.899438 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx"] Nov 01 00:46:01.906874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.906829 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz"] Nov 01 00:46:01.947285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947248 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.947285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947282 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.947556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947320 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.947556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947426 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.947556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947498 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.947556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947542 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.947843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947568 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.947843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947615 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.947843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947678 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55965aa-c817-4645-a465-6ba90a23486d-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.947843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947780 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.947843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947811 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.948127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947939 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55965aa-c817-4645-a465-6ba90a23486d-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.948127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.947999 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxv6z\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-kube-api-access-fxv6z\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.948127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.948034 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:01.948127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.948083 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.948333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.948157 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.948333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.948201 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.948333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.948230 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66q8s\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-kube-api-access-66q8s\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:01.951305 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:01.951280 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-30fb19-openshift-default-dockercfg-tm5sb\"" Nov 01 00:46:02.049424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049336 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.049664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049617 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55965aa-c817-4645-a465-6ba90a23486d-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.049766 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.049766 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049730 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.049876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049787 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55965aa-c817-4645-a465-6ba90a23486d-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.049876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049822 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxv6z\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-kube-api-access-fxv6z\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.049876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049855 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.050131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049910 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.050131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049959 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.050131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.049999 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.050131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66q8s\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-kube-api-access-66q8s\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.050131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050074 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.050131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050105 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.050441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050154 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.050441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050184 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.050441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050207 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.050596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050466 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.051001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050930 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.051133 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.051051 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55965aa-c817-4645-a465-6ba90a23486d-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.051700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.050214 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.051798 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.051765 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.051855 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.051801 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.051914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.051864 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.052300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.052275 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.052393 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.052358 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.052555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.052531 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.052873 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.052847 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.053474 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.053417 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.053716 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.053694 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55965aa-c817-4645-a465-6ba90a23486d-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.054771 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.054742 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.055288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.055262 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.055423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.055405 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.112573 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.112538 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.113074 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.113051 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.113230 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.113211 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxv6z\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-kube-api-access-fxv6z\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.113303 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.113280 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66q8s\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-kube-api-access-66q8s\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.115064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.115045 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:02.164356 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.164323 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:02.393047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.392967 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-fvzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 00:46:02.393047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.393021 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 00:46:02.566041 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:46:02.566007 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55965aa_c817_4645_a465_6ba90a23486d.slice/crio-b24c15a63f092e58c98ef81334ac9ce445744e9844af8b4650c6aa85449937c9 WatchSource:0}: Error finding container b24c15a63f092e58c98ef81334ac9ce445744e9844af8b4650c6aa85449937c9: Status 404 returned error can't find the container with id b24c15a63f092e58c98ef81334ac9ce445744e9844af8b4650c6aa85449937c9 Nov 01 00:46:02.568978 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.568954 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx"] Nov 01 00:46:02.569097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.568970 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:02.569097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.569046 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:02.569097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.569092 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:02.613519 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:46:02.613474 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15803f9b_8f6f_4169_8e00_4b62fcc1e09d.slice/crio-49569a0d4714d8ac86493fa1d525b0c2b6942d58d5364c9b2eea7c57f735c552 WatchSource:0}: Error finding container 49569a0d4714d8ac86493fa1d525b0c2b6942d58d5364c9b2eea7c57f735c552: Status 404 returned error can't find the container with id 49569a0d4714d8ac86493fa1d525b0c2b6942d58d5364c9b2eea7c57f735c552 Nov 01 00:46:02.613848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.613824 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz"] Nov 01 00:46:02.616326 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.616290 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:02.616426 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.616373 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:02.616426 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:02.616420 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:46:03.448644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.448562 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" event={"ID":"15803f9b-8f6f-4169-8e00-4b62fcc1e09d","Type":"ContainerStarted","Data":"676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2"} Nov 01 00:46:03.449020 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.448655 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" event={"ID":"15803f9b-8f6f-4169-8e00-4b62fcc1e09d","Type":"ContainerStarted","Data":"49569a0d4714d8ac86493fa1d525b0c2b6942d58d5364c9b2eea7c57f735c552"} Nov 01 00:46:03.451806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.451779 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" event={"ID":"f55965aa-c817-4645-a465-6ba90a23486d","Type":"ContainerStarted","Data":"32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613"} Nov 01 00:46:03.451997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.451982 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" event={"ID":"f55965aa-c817-4645-a465-6ba90a23486d","Type":"ContainerStarted","Data":"b24c15a63f092e58c98ef81334ac9ce445744e9844af8b4650c6aa85449937c9"} Nov 01 00:46:03.591648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.590717 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podStartSLOduration=2.590695787 podStartE2EDuration="2.590695787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:03.586969823 +0000 UTC m=+1353.566552158" watchObservedRunningTime="2026-11-01 00:46:03.590695787 +0000 UTC m=+1353.570278123" Nov 01 00:46:03.725703 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.718509 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podStartSLOduration=2.718486558 podStartE2EDuration="2.718486558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:03.716084038 +0000 UTC m=+1353.695666380" watchObservedRunningTime="2026-11-01 00:46:03.718486558 +0000 UTC m=+1353.698068901" Nov 01 00:46:03.906651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.906069 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:46:03.907646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.907601 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:46:03.908997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:03.908977 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:46:04.117858 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:04.116860 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:04.118066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:04.118010 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 00:46:04.118145 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:04.118065 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 00:46:04.175674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:04.175423 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:04.175674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:04.175553 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 00:46:04.181647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:04.175611 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 00:46:05.116161 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:05.116123 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 00:46:05.116581 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:05.116190 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 00:46:05.167650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:05.167036 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 00:46:05.167650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:05.167112 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 00:46:06.116246 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:06.116213 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 00:46:06.116759 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:06.116283 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 00:46:06.173105 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:06.172985 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 00:46:06.173105 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:06.173050 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 00:46:07.116278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:07.116232 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 00:46:07.116733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:07.116300 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 00:46:07.168070 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:07.168031 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 00:46:07.168369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:07.168328 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 00:46:08.116230 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:08.116195 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 00:46:08.116444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:08.116255 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 00:46:08.172389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:08.172353 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 00:46:08.173213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:08.173179 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 00:46:09.116183 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:09.116150 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 00:46:09.116353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:09.116211 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 00:46:09.164903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:09.164869 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 01 00:46:09.165280 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:09.164916 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 01 00:46:10.120181 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:10.120151 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:10.120678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:10.120657 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:10.121840 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:10.121817 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 00:46:10.170795 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:10.170764 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:10.171143 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:10.171043 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:10.171971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:10.171950 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:46:13.102358 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:46:13.102321 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:46:19.323060 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.323029 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742"] Nov 01 00:46:19.327144 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.327118 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.430888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.430846 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"kube-root-ca.crt\"" Nov 01 00:46:19.430888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.430878 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"default-dockercfg-d59kz\"" Nov 01 00:46:19.431129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.430918 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"openshift-service-ca.crt\"" Nov 01 00:46:19.431129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.430938 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-32439f\"/\"distinct-w0-32439f-kserve-self-signed-certs\"" Nov 01 00:46:19.435138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.435116 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742"] Nov 01 00:46:19.445259 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.445234 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-dshm\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.445410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.445271 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tls-certs\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.445410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.445306 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dv9\" (UniqueName: \"kubernetes.io/projected/e9b5e5da-310d-469f-b7b3-cd6f08314deb-kube-api-access-52dv9\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.445531 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.445498 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tmp-dir\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.445611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.445597 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-model-cache\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.445713 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.445654 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-home\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547224 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-model-cache\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547278 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-home\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547395 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-dshm\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547420 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tls-certs\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547450 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dv9\" (UniqueName: \"kubernetes.io/projected/e9b5e5da-310d-469f-b7b3-cd6f08314deb-kube-api-access-52dv9\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547714 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547545 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tmp-dir\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547731 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-model-cache\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547763 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-home\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.547937 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.547918 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tmp-dir\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.550164 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.550122 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-dshm\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.550287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.550171 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tls-certs\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.609662 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.609554 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dv9\" (UniqueName: \"kubernetes.io/projected/e9b5e5da-310d-469f-b7b3-cd6f08314deb-kube-api-access-52dv9\") pod \"distinct-w0-32439f-kserve-76c8d8dc7d-v4742\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:19.641764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:19.641725 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:20.031617 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:46:20.031583 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9b5e5da_310d_469f_b7b3_cd6f08314deb.slice/crio-80be61381a63bca8113ff198e7fd420e31f32f90757221422f6d682aa9c14e02 WatchSource:0}: Error finding container 80be61381a63bca8113ff198e7fd420e31f32f90757221422f6d682aa9c14e02: Status 404 returned error can't find the container with id 80be61381a63bca8113ff198e7fd420e31f32f90757221422f6d682aa9c14e02 Nov 01 00:46:20.031788 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:20.031677 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742"] Nov 01 00:46:20.643991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:20.643958 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" event={"ID":"e9b5e5da-310d-469f-b7b3-cd6f08314deb","Type":"ContainerStarted","Data":"6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12"} Nov 01 00:46:20.644414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:20.643999 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" event={"ID":"e9b5e5da-310d-469f-b7b3-cd6f08314deb","Type":"ContainerStarted","Data":"80be61381a63bca8113ff198e7fd420e31f32f90757221422f6d682aa9c14e02"} Nov 01 00:46:20.763322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:20.763269 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podStartSLOduration=1.763254068 podStartE2EDuration="1.763254068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:46:20.760911818 +0000 UTC m=+1370.740494152" watchObservedRunningTime="2026-11-01 00:46:20.763254068 +0000 UTC m=+1370.742836403" Nov 01 00:46:21.642285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:21.642247 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:29.642162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:29.642126 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:29.654909 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:29.654887 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:29.746125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:46:29.746099 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:46:43.103088 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:46:43.103050 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:47:13.106169 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:47:13.106126 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:47:13.421608 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.421528 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt"] Nov 01 00:47:13.425847 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.425832 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.477398 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.477363 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tmp-dir\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.477596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.477439 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-home\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.477596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.477501 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tls-certs\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.477790 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.477767 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-model-cache\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.477875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.477806 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwss\" (UniqueName: \"kubernetes.io/projected/a54f758e-a86e-44e8-aae6-eef45c5e6dda-kube-api-access-bdwss\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.477933 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.477892 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-dshm\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.529436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.529403 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-43a0cf\"/\"distinct-2-w4-43a0cf-kserve-self-signed-certs\"" Nov 01 00:47:13.534098 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.534071 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt"] Nov 01 00:47:13.586650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.581601 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-model-cache\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.586650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.585718 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwss\" (UniqueName: \"kubernetes.io/projected/a54f758e-a86e-44e8-aae6-eef45c5e6dda-kube-api-access-bdwss\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.586650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.585834 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-dshm\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.586650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.585883 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tmp-dir\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.586650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.585951 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-home\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.586650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.585973 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tls-certs\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.594215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.582091 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-model-cache\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.594667 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.594604 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-home\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.594872 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.594852 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tmp-dir\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.597644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.597598 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tls-certs\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.598235 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.598205 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-dshm\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.655989 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.655964 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwss\" (UniqueName: \"kubernetes.io/projected/a54f758e-a86e-44e8-aae6-eef45c5e6dda-kube-api-access-bdwss\") pod \"distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:13.738979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:13.738887 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:14.183102 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:47:14.183074 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda54f758e_a86e_44e8_aae6_eef45c5e6dda.slice/crio-60f7d4d76ef63befba9aaba7f2c05c1c797b976ca3461ef109c606ec1e63d051 WatchSource:0}: Error finding container 60f7d4d76ef63befba9aaba7f2c05c1c797b976ca3461ef109c606ec1e63d051: Status 404 returned error can't find the container with id 60f7d4d76ef63befba9aaba7f2c05c1c797b976ca3461ef109c606ec1e63d051 Nov 01 00:47:14.183527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:14.183506 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt"] Nov 01 00:47:15.093840 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:15.093796 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" event={"ID":"a54f758e-a86e-44e8-aae6-eef45c5e6dda","Type":"ContainerStarted","Data":"6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365"} Nov 01 00:47:15.094015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:15.093851 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" event={"ID":"a54f758e-a86e-44e8-aae6-eef45c5e6dda","Type":"ContainerStarted","Data":"60f7d4d76ef63befba9aaba7f2c05c1c797b976ca3461ef109c606ec1e63d051"} Nov 01 00:47:15.214515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:15.214460 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podStartSLOduration=2.214447515 podStartE2EDuration="2.214447515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:15.210318792 +0000 UTC m=+1425.189901128" watchObservedRunningTime="2026-11-01 00:47:15.214447515 +0000 UTC m=+1425.194029867" Nov 01 00:47:15.739740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:15.739703 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:20.782997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.782965 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth"] Nov 01 00:47:20.787096 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.787081 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875321 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875279 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875334 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875477 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875511 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875541 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbce44-5517-4767-9abf-d13454fa9148-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875832 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875801 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbce44-5517-4767-9abf-d13454fa9148-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875881 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875849 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhn9\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-kube-api-access-xxhn9\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.875997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.875982 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.876032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.876012 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.891255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.891223 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth"] Nov 01 00:47:20.977512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977474 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.977512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977517 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.977780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977654 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbce44-5517-4767-9abf-d13454fa9148-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.977780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977757 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbce44-5517-4767-9abf-d13454fa9148-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.977780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977776 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn9\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-kube-api-access-xxhn9\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.977938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977838 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.977938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977858 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.977938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977909 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.978095 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977937 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.978095 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.977951 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.978209 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.978193 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.978276 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.978222 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.978382 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.978360 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbce44-5517-4767-9abf-d13454fa9148-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.978572 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.978551 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.979917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.979890 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:20.980358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:20.980332 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbce44-5517-4767-9abf-d13454fa9148-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:21.039045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:21.038976 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:21.039179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:21.039158 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhn9\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-kube-api-access-xxhn9\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:21.102330 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:21.102300 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:21.537382 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:47:21.537350 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dbce44_5517_4767_9abf_d13454fa9148.slice/crio-c2633b9c104df8fce0aaf400ff3e52f4a1dc020d037033efd7bc1d69d37dd824 WatchSource:0}: Error finding container c2633b9c104df8fce0aaf400ff3e52f4a1dc020d037033efd7bc1d69d37dd824: Status 404 returned error can't find the container with id c2633b9c104df8fce0aaf400ff3e52f4a1dc020d037033efd7bc1d69d37dd824 Nov 01 00:47:21.538406 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:21.538383 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth"] Nov 01 00:47:21.539736 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:21.539704 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:21.539813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:21.539770 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:21.539813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:21.539799 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:22.146461 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:22.146425 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" event={"ID":"28dbce44-5517-4767-9abf-d13454fa9148","Type":"ContainerStarted","Data":"af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627"} Nov 01 00:47:22.146967 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:22.146477 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" event={"ID":"28dbce44-5517-4767-9abf-d13454fa9148","Type":"ContainerStarted","Data":"c2633b9c104df8fce0aaf400ff3e52f4a1dc020d037033efd7bc1d69d37dd824"} Nov 01 00:47:22.268276 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:22.268226 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podStartSLOduration=2.268212541 podStartE2EDuration="2.268212541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:22.266941703 +0000 UTC m=+1432.246524048" watchObservedRunningTime="2026-11-01 00:47:22.268212541 +0000 UTC m=+1432.247794877" Nov 01 00:47:23.102925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:23.102889 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:23.104529 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:23.104505 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 00:47:23.104668 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:23.104554 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 00:47:23.740117 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:23.740084 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:23.782493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:23.782457 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:24.103367 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:24.103331 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 00:47:24.103560 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:24.103385 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 00:47:24.198165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:24.198134 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:47:25.103128 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:25.103091 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 00:47:25.103572 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:25.103147 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 00:47:26.103231 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:26.103197 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 00:47:26.103794 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:26.103248 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 00:47:27.102747 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:27.102716 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 00:47:27.103014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:27.102976 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 00:47:28.103050 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:28.103004 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 00:47:28.103470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:28.103080 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 00:47:29.108100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:29.108067 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:29.108609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:29.108589 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:29.109393 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:29.109374 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 00:47:31.792444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.792408 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26"] Nov 01 00:47:31.797722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.797701 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.827761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.827731 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.827883 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.827771 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05687ed-54f9-4637-a3d6-451b174895e4-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.827957 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.827937 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.827994 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.827986 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzm6q\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-kube-api-access-fzm6q\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.828044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.828012 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.828110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.828094 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.828198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.828182 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.828310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.828294 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.828368 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.828328 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05687ed-54f9-4637-a3d6-451b174895e4-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.907296 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.907260 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26"] Nov 01 00:47:31.929559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.929816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929567 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05687ed-54f9-4637-a3d6-451b174895e4-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.929816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929651 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.929816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929684 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05687ed-54f9-4637-a3d6-451b174895e4-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.929816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929772 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.930057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929826 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzm6q\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-kube-api-access-fzm6q\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.930057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929863 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.930057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929890 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.930057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.929931 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.930455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.930429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.931021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.930994 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.931284 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.931258 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.931528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.931507 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.932356 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.932330 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05687ed-54f9-4637-a3d6-451b174895e4-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.933175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.933147 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.934128 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.934102 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05687ed-54f9-4637-a3d6-451b174895e4-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.991740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.991706 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:31.991932 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:31.991908 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzm6q\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-kube-api-access-fzm6q\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:32.113074 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:32.112993 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:32.555383 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:47:32.555354 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05687ed_54f9_4637_a3d6_451b174895e4.slice/crio-6725b7b28300977ced466887e2f4ed55d84d913f998899a4257f49538ad86d46 WatchSource:0}: Error finding container 6725b7b28300977ced466887e2f4ed55d84d913f998899a4257f49538ad86d46: Status 404 returned error can't find the container with id 6725b7b28300977ced466887e2f4ed55d84d913f998899a4257f49538ad86d46 Nov 01 00:47:32.555521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:32.555460 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26"] Nov 01 00:47:32.557786 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:32.557753 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:32.557880 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:32.557815 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:32.557880 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:32.557848 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:33.283223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:33.283187 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" event={"ID":"a05687ed-54f9-4637-a3d6-451b174895e4","Type":"ContainerStarted","Data":"58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f"} Nov 01 00:47:33.283667 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:33.283246 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" event={"ID":"a05687ed-54f9-4637-a3d6-451b174895e4","Type":"ContainerStarted","Data":"6725b7b28300977ced466887e2f4ed55d84d913f998899a4257f49538ad86d46"} Nov 01 00:47:33.506333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:33.506272 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podStartSLOduration=2.5062569850000003 podStartE2EDuration="2.506256985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:33.454094884 +0000 UTC m=+1443.433677221" watchObservedRunningTime="2026-11-01 00:47:33.506256985 +0000 UTC m=+1443.485839320" Nov 01 00:47:34.113489 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:34.113453 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:34.115035 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:34.115013 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 00:47:34.115150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:34.115062 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 00:47:35.113796 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:35.113758 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 00:47:35.114276 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:35.113811 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 00:47:36.113902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:36.113867 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 00:47:36.114343 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:36.113938 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 00:47:37.113386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:37.113350 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 00:47:37.113576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:37.113414 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 00:47:38.113424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:38.113391 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 00:47:38.113877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:38.113455 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 00:47:39.113369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:39.113333 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 00:47:39.113562 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:39.113403 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 00:47:40.119045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:40.119017 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:40.119468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:40.119453 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:40.120257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:40.120237 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:47:43.104665 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:47:43.104590 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:47:49.799669 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.799610 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc"] Nov 01 00:47:49.803750 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.803729 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.910554 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.910484 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc"] Nov 01 00:47:49.988003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.987968 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988009 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988090 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988112 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfsq\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-kube-api-access-tzfsq\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988264 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988290 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988407 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988390 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988456 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:49.988566 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:49.988548 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090011 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090051 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090098 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090157 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090228 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090370 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090475 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090429 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090480 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090506 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090520 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfsq\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-kube-api-access-tzfsq\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090565 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.090877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090850 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.091013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.090995 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.092806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.092779 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.093093 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.093063 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.153322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.153289 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.153493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.153435 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfsq\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-kube-api-access-tzfsq\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.421348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.421322 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:50.865144 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:47:50.865115 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20b65f7_9b42_4b6f_8a59_5eb5a0d13af4.slice/crio-474cef4f3481942d9bb86fe80a7d4d49789a3bd793bd88ebd5d95f431ae1ef0f WatchSource:0}: Error finding container 474cef4f3481942d9bb86fe80a7d4d49789a3bd793bd88ebd5d95f431ae1ef0f: Status 404 returned error can't find the container with id 474cef4f3481942d9bb86fe80a7d4d49789a3bd793bd88ebd5d95f431ae1ef0f Nov 01 00:47:50.865470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.865404 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc"] Nov 01 00:47:50.867923 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.867884 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:50.868024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.867970 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:50.868024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:50.868012 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:47:51.481786 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:51.481742 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" event={"ID":"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4","Type":"ContainerStarted","Data":"a7203722ce13cde00c55bd2dbebe9079b5e4b4b2aeee2bd7a8e9422a0738a191"} Nov 01 00:47:51.481973 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:51.481806 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" event={"ID":"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4","Type":"ContainerStarted","Data":"474cef4f3481942d9bb86fe80a7d4d49789a3bd793bd88ebd5d95f431ae1ef0f"} Nov 01 00:47:51.604266 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:51.604206 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podStartSLOduration=2.604191287 podStartE2EDuration="2.604191287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:47:51.60186176 +0000 UTC m=+1461.581444095" watchObservedRunningTime="2026-11-01 00:47:51.604191287 +0000 UTC m=+1461.583773622" Nov 01 00:47:52.422235 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:52.422197 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:52.423656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:52.423616 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 00:47:52.423783 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:52.423680 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 00:47:53.422667 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:53.422615 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 00:47:53.423073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:53.422685 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 00:47:54.422566 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:54.422533 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 00:47:54.422765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:54.422589 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 00:47:55.422449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:55.422411 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 00:47:55.422670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:55.422470 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 00:47:56.422325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:56.422293 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 00:47:56.422758 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:56.422348 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 00:47:57.421970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:57.421928 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 00:47:57.422167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:57.421993 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 00:47:58.428453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:58.428420 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:58.428993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:58.428879 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:47:58.429654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:47:58.429607 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 00:48:01.002981 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.002946 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6"] Nov 01 00:48:01.008254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.008236 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.059670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.059638 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc495\" (UniqueName: \"kubernetes.io/projected/682b0f69-29fc-4a05-a330-9c24b4371708-kube-api-access-hc495\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.059829 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.059735 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.059894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.059842 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.063975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.063957 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6"] Nov 01 00:48:01.109602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.109566 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:48:01.109602 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.109574 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:48:01.161038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.161004 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc495\" (UniqueName: \"kubernetes.io/projected/682b0f69-29fc-4a05-a330-9c24b4371708-kube-api-access-hc495\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.161223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.161048 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.161223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.161087 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.161769 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.161741 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.163662 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.163609 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.222416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.222390 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc495\" (UniqueName: \"kubernetes.io/projected/682b0f69-29fc-4a05-a330-9c24b4371708-kube-api-access-hc495\") pod \"maas-api-revoke-sub-304bd02a-fkkk6\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.322339 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.322256 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:01.706430 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:48:01.706399 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682b0f69_29fc_4a05_a330_9c24b4371708.slice/crio-288d0fea12b074a5a3403348fe68ee289eaacecad3ac24703018154fd0574cf9 WatchSource:0}: Error finding container 288d0fea12b074a5a3403348fe68ee289eaacecad3ac24703018154fd0574cf9: Status 404 returned error can't find the container with id 288d0fea12b074a5a3403348fe68ee289eaacecad3ac24703018154fd0574cf9 Nov 01 00:48:01.707509 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:01.707485 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6"] Nov 01 00:48:02.591423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:02.591385 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" event={"ID":"682b0f69-29fc-4a05-a330-9c24b4371708","Type":"ContainerStarted","Data":"0bc30325aed8aae40f13eef7852342ef63e76e19cf1ec39c8e4cb38fa57f56c9"} Nov 01 00:48:02.591853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:02.591444 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" event={"ID":"682b0f69-29fc-4a05-a330-9c24b4371708","Type":"ContainerStarted","Data":"288d0fea12b074a5a3403348fe68ee289eaacecad3ac24703018154fd0574cf9"} Nov 01 00:48:02.711619 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:02.711567 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" podStartSLOduration=2.7115527139999998 podStartE2EDuration="2.711552714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:48:02.707500968 +0000 UTC m=+1472.687083304" watchObservedRunningTime="2026-11-01 00:48:02.711552714 +0000 UTC m=+1472.691135048" Nov 01 00:48:03.601460 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:03.601424 4910 generic.go:356] "Generic (PLEG): container finished" podID="682b0f69-29fc-4a05-a330-9c24b4371708" containerID="0bc30325aed8aae40f13eef7852342ef63e76e19cf1ec39c8e4cb38fa57f56c9" exitCode=0 Nov 01 00:48:03.601937 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:03.601505 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" event={"ID":"682b0f69-29fc-4a05-a330-9c24b4371708","Type":"ContainerDied","Data":"0bc30325aed8aae40f13eef7852342ef63e76e19cf1ec39c8e4cb38fa57f56c9"} Nov 01 00:48:04.774771 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.774745 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:04.809837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.809802 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-openshift-ca\") pod \"682b0f69-29fc-4a05-a330-9c24b4371708\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " Nov 01 00:48:04.809998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.809918 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682b0f69-29fc-4a05-a330-9c24b4371708-kube-api-access-hc495\" (UniqueName: \"kubernetes.io/projected/682b0f69-29fc-4a05-a330-9c24b4371708-kube-api-access-hc495\") pod \"682b0f69-29fc-4a05-a330-9c24b4371708\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " Nov 01 00:48:04.810060 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.810022 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-xks-ca\") pod \"682b0f69-29fc-4a05-a330-9c24b4371708\" (UID: \"682b0f69-29fc-4a05-a330-9c24b4371708\") " Nov 01 00:48:04.810442 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.810352 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-openshift-ca" pod "682b0f69-29fc-4a05-a330-9c24b4371708" (UID: "682b0f69-29fc-4a05-a330-9c24b4371708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:48:04.810735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.810713 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:04.812230 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.812206 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-xks-ca" pod "682b0f69-29fc-4a05-a330-9c24b4371708" (UID: "682b0f69-29fc-4a05-a330-9c24b4371708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:48:04.812493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.812468 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682b0f69-29fc-4a05-a330-9c24b4371708-kube-api-access-hc495" pod "682b0f69-29fc-4a05-a330-9c24b4371708" (UID: "682b0f69-29fc-4a05-a330-9c24b4371708"). InnerVolumeSpecName "kube-api-access-hc495". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:48:04.912163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.912084 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682b0f69-29fc-4a05-a330-9c24b4371708-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:04.912163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:04.912112 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc495\" (UniqueName: \"kubernetes.io/projected/682b0f69-29fc-4a05-a330-9c24b4371708-kube-api-access-hc495\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:05.620515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:05.620488 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" Nov 01 00:48:05.620781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:05.620528 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6" event={"ID":"682b0f69-29fc-4a05-a330-9c24b4371708","Type":"ContainerDied","Data":"288d0fea12b074a5a3403348fe68ee289eaacecad3ac24703018154fd0574cf9"} Nov 01 00:48:05.620781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:05.620587 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288d0fea12b074a5a3403348fe68ee289eaacecad3ac24703018154fd0574cf9" Nov 01 00:48:13.105232 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:48:13.105195 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:48:30.519800 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:30.519758 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:48:30.522968 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:30.522946 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:48:30.530712 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:30.530689 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:48:30.533430 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:30.533408 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:48:31.828846 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.828812 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc"] Nov 01 00:48:31.829369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.829355 4910 cpu_manager.go:414] "containerMap: removing container" podUID="682b0f69-29fc-4a05-a330-9c24b4371708" containerName="revoke-keys" Nov 01 00:48:31.829413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.829370 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682b0f69-29fc-4a05-a330-9c24b4371708" containerName="revoke-keys" Nov 01 00:48:31.829487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.829479 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="682b0f69-29fc-4a05-a330-9c24b4371708" containerName="revoke-keys" Nov 01 00:48:31.833147 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.833131 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:31.891197 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.891165 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:48:31.919793 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.919758 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:31.920073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.920054 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcp8w\" (UniqueName: \"kubernetes.io/projected/c850074f-db5e-4858-9d41-e03da64160c9-kube-api-access-xcp8w\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:31.920258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.920234 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:31.940874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.940851 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:48:31.949167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:31.949146 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc"] Nov 01 00:48:32.021532 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.021502 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:32.021687 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.021564 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcp8w\" (UniqueName: \"kubernetes.io/projected/c850074f-db5e-4858-9d41-e03da64160c9-kube-api-access-xcp8w\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:32.021687 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.021617 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:32.022215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.022191 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:32.024189 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.024163 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:32.083729 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.083670 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcp8w\" (UniqueName: \"kubernetes.io/projected/c850074f-db5e-4858-9d41-e03da64160c9-kube-api-access-xcp8w\") pod \"maas-api-revoke-sub-974ade6e-952bc\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:32.146735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.146708 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:32.535667 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.535639 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc"] Nov 01 00:48:32.537472 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:48:32.537442 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc850074f_db5e_4858_9d41_e03da64160c9.slice/crio-79ead7e5e027e690ded295da73dce815d859e6df46604b611594ec08c5c84e7c WatchSource:0}: Error finding container 79ead7e5e027e690ded295da73dce815d859e6df46604b611594ec08c5c84e7c: Status 404 returned error can't find the container with id 79ead7e5e027e690ded295da73dce815d859e6df46604b611594ec08c5c84e7c Nov 01 00:48:32.859114 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:32.859082 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" event={"ID":"c850074f-db5e-4858-9d41-e03da64160c9","Type":"ContainerStarted","Data":"79ead7e5e027e690ded295da73dce815d859e6df46604b611594ec08c5c84e7c"} Nov 01 00:48:33.872235 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:33.872185 4910 generic.go:356] "Generic (PLEG): container finished" podID="c850074f-db5e-4858-9d41-e03da64160c9" containerID="8479ca7a671dff3893edcff724667a38bd6c5c6ea2d58df0e2a05c9d5fe8813a" exitCode=0 Nov 01 00:48:33.872684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:33.872257 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" event={"ID":"c850074f-db5e-4858-9d41-e03da64160c9","Type":"ContainerDied","Data":"8479ca7a671dff3893edcff724667a38bd6c5c6ea2d58df0e2a05c9d5fe8813a"} Nov 01 00:48:35.040377 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.040352 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:35.065551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.065526 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-xks-ca\") pod \"c850074f-db5e-4858-9d41-e03da64160c9\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " Nov 01 00:48:35.065700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.065585 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c850074f-db5e-4858-9d41-e03da64160c9-kube-api-access-xcp8w\" (UniqueName: \"kubernetes.io/projected/c850074f-db5e-4858-9d41-e03da64160c9-kube-api-access-xcp8w\") pod \"c850074f-db5e-4858-9d41-e03da64160c9\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " Nov 01 00:48:35.065700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.065682 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-openshift-ca\") pod \"c850074f-db5e-4858-9d41-e03da64160c9\" (UID: \"c850074f-db5e-4858-9d41-e03da64160c9\") " Nov 01 00:48:35.066089 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.066057 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-openshift-ca" pod "c850074f-db5e-4858-9d41-e03da64160c9" (UID: "c850074f-db5e-4858-9d41-e03da64160c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:48:35.066375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.066251 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:35.067780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.067757 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c850074f-db5e-4858-9d41-e03da64160c9-kube-api-access-xcp8w" pod "c850074f-db5e-4858-9d41-e03da64160c9" (UID: "c850074f-db5e-4858-9d41-e03da64160c9"). InnerVolumeSpecName "kube-api-access-xcp8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:48:35.067878 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.067853 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-xks-ca" pod "c850074f-db5e-4858-9d41-e03da64160c9" (UID: "c850074f-db5e-4858-9d41-e03da64160c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:48:35.167177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.167083 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c850074f-db5e-4858-9d41-e03da64160c9-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:35.167177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.167119 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcp8w\" (UniqueName: \"kubernetes.io/projected/c850074f-db5e-4858-9d41-e03da64160c9-kube-api-access-xcp8w\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:35.900343 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.900310 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" event={"ID":"c850074f-db5e-4858-9d41-e03da64160c9","Type":"ContainerDied","Data":"79ead7e5e027e690ded295da73dce815d859e6df46604b611594ec08c5c84e7c"} Nov 01 00:48:35.900515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.900354 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc" Nov 01 00:48:35.900515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:35.900363 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79ead7e5e027e690ded295da73dce815d859e6df46604b611594ec08c5c84e7c" Nov 01 00:48:36.041410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.041379 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz"] Nov 01 00:48:36.041966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.041951 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c850074f-db5e-4858-9d41-e03da64160c9" containerName="revoke-keys" Nov 01 00:48:36.042011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.041967 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c850074f-db5e-4858-9d41-e03da64160c9" containerName="revoke-keys" Nov 01 00:48:36.042085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.042076 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c850074f-db5e-4858-9d41-e03da64160c9" containerName="revoke-keys" Nov 01 00:48:36.045860 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.045837 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077429 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077603 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077467 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077603 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077542 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077603 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077587 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vkfn\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-kube-api-access-9vkfn\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077605 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077667 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077708 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077729 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.077903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.077828 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.098891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.098855 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5a9c19-openshift-default-dockercfg-vrnmb\"" Nov 01 00:48:36.154438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.154364 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz"] Nov 01 00:48:36.178753 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.178719 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.178913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.178771 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vkfn\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-kube-api-access-9vkfn\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.178913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.178801 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.178913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.178850 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.179081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.178930 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.179081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.178962 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.179081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.179036 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.179250 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.179125 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.179250 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.179169 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.180447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.179519 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.180447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.179550 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.180447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.179690 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.180447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.180125 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.180447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.180404 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.182684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.182557 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.182684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.182603 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.241341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.241307 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.241462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.241365 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vkfn\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-kube-api-access-9vkfn\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.359761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.359730 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:36.802069 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.802043 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz"] Nov 01 00:48:36.802476 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:48:36.802443 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfdd122_d2a5_4e1f_8955_a64b56cf435a.slice/crio-6813a404bac7bab6937341535fc9b65a876f8eed0ec0a038a030f9a717a959fb WatchSource:0}: Error finding container 6813a404bac7bab6937341535fc9b65a876f8eed0ec0a038a030f9a717a959fb: Status 404 returned error can't find the container with id 6813a404bac7bab6937341535fc9b65a876f8eed0ec0a038a030f9a717a959fb Nov 01 00:48:36.805114 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.805085 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:48:36.805180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.805153 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:48:36.805215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.805181 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:48:36.911697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.911652 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" event={"ID":"acfdd122-d2a5-4e1f-8955-a64b56cf435a","Type":"ContainerStarted","Data":"454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185"} Nov 01 00:48:36.911867 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:36.911727 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" event={"ID":"acfdd122-d2a5-4e1f-8955-a64b56cf435a","Type":"ContainerStarted","Data":"6813a404bac7bab6937341535fc9b65a876f8eed0ec0a038a030f9a717a959fb"} Nov 01 00:48:37.037303 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:37.037252 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podStartSLOduration=2.037237649 podStartE2EDuration="2.037237649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:48:37.034122856 +0000 UTC m=+1507.013705194" watchObservedRunningTime="2026-11-01 00:48:37.037237649 +0000 UTC m=+1507.016819983" Nov 01 00:48:37.360692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:37.360660 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:37.362695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:37.362662 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 01 00:48:37.362833 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:37.362713 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 01 00:48:38.366650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:38.361046 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 01 00:48:38.366650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:38.361107 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 01 00:48:39.366651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:39.365723 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 01 00:48:39.366651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:39.365791 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 01 00:48:40.367650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:40.367347 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 01 00:48:40.367650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:40.367403 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 01 00:48:41.365651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:41.360958 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 01 00:48:41.365651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:41.361015 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 01 00:48:42.374780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:42.374743 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 01 00:48:42.375233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:42.374812 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 01 00:48:43.106104 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:48:43.106062 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:48:43.360227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:43.360130 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 01 00:48:43.360227 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:43.360194 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 01 00:48:44.425951 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:44.425903 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:44.426709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:44.426687 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:44.427738 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:44.427715 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:48:47.486680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.486647 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx"] Nov 01 00:48:47.491053 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.491035 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.593781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.593752 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:48:47.593781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.593770 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:48:47.598452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.598427 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx"] Nov 01 00:48:47.649708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.649675 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.649873 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.649729 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.649873 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.649831 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjlf\" (UniqueName: \"kubernetes.io/projected/ae290517-e58a-48ff-8e28-495f3ce917e3-kube-api-access-9vjlf\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.751390 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.751367 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.751551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.751421 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.751551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.751459 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjlf\" (UniqueName: \"kubernetes.io/projected/ae290517-e58a-48ff-8e28-495f3ce917e3-kube-api-access-9vjlf\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.752024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.752002 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.753959 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.753930 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:47.813175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:47.813146 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjlf\" (UniqueName: \"kubernetes.io/projected/ae290517-e58a-48ff-8e28-495f3ce917e3-kube-api-access-9vjlf\") pod \"maas-api-revoke-sub-9efe04b8-nwttx\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:48.105413 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:48.105341 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:48.488103 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:48:48.488064 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae290517_e58a_48ff_8e28_495f3ce917e3.slice/crio-2c44fcc7b1ed1c94e3b28415fd62d591c082ec321eb23929bc578a438542d8ad WatchSource:0}: Error finding container 2c44fcc7b1ed1c94e3b28415fd62d591c082ec321eb23929bc578a438542d8ad: Status 404 returned error can't find the container with id 2c44fcc7b1ed1c94e3b28415fd62d591c082ec321eb23929bc578a438542d8ad Nov 01 00:48:48.488841 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:48.488820 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx"] Nov 01 00:48:49.032236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:49.032140 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" event={"ID":"ae290517-e58a-48ff-8e28-495f3ce917e3","Type":"ContainerStarted","Data":"5db3e12de71970ffc77f96706596a331758af31237283e7545bbad3ede15fdb0"} Nov 01 00:48:49.032236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:49.032215 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" event={"ID":"ae290517-e58a-48ff-8e28-495f3ce917e3","Type":"ContainerStarted","Data":"2c44fcc7b1ed1c94e3b28415fd62d591c082ec321eb23929bc578a438542d8ad"} Nov 01 00:48:49.152500 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:49.152446 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" podStartSLOduration=2.152431463 podStartE2EDuration="2.152431463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:48:49.151770341 +0000 UTC m=+1519.131352682" watchObservedRunningTime="2026-11-01 00:48:49.152431463 +0000 UTC m=+1519.132013803" Nov 01 00:48:50.041476 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:50.041443 4910 generic.go:356] "Generic (PLEG): container finished" podID="ae290517-e58a-48ff-8e28-495f3ce917e3" containerID="5db3e12de71970ffc77f96706596a331758af31237283e7545bbad3ede15fdb0" exitCode=0 Nov 01 00:48:50.041961 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:50.041538 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" event={"ID":"ae290517-e58a-48ff-8e28-495f3ce917e3","Type":"ContainerDied","Data":"5db3e12de71970ffc77f96706596a331758af31237283e7545bbad3ede15fdb0"} Nov 01 00:48:51.203491 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.203468 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:51.300646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.300550 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-xks-ca\") pod \"ae290517-e58a-48ff-8e28-495f3ce917e3\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " Nov 01 00:48:51.300802 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.300676 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-openshift-ca\") pod \"ae290517-e58a-48ff-8e28-495f3ce917e3\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " Nov 01 00:48:51.300802 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.300775 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae290517-e58a-48ff-8e28-495f3ce917e3-kube-api-access-9vjlf\" (UniqueName: \"kubernetes.io/projected/ae290517-e58a-48ff-8e28-495f3ce917e3-kube-api-access-9vjlf\") pod \"ae290517-e58a-48ff-8e28-495f3ce917e3\" (UID: \"ae290517-e58a-48ff-8e28-495f3ce917e3\") " Nov 01 00:48:51.301046 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.301024 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-openshift-ca" pod "ae290517-e58a-48ff-8e28-495f3ce917e3" (UID: "ae290517-e58a-48ff-8e28-495f3ce917e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:48:51.301375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.301325 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:51.303015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.302995 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-xks-ca" pod "ae290517-e58a-48ff-8e28-495f3ce917e3" (UID: "ae290517-e58a-48ff-8e28-495f3ce917e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:48:51.303097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.303009 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae290517-e58a-48ff-8e28-495f3ce917e3-kube-api-access-9vjlf" pod "ae290517-e58a-48ff-8e28-495f3ce917e3" (UID: "ae290517-e58a-48ff-8e28-495f3ce917e3"). InnerVolumeSpecName "kube-api-access-9vjlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:48:51.402104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.402075 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vjlf\" (UniqueName: \"kubernetes.io/projected/ae290517-e58a-48ff-8e28-495f3ce917e3-kube-api-access-9vjlf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:51.402104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:51.402104 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae290517-e58a-48ff-8e28-495f3ce917e3-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:48:52.083885 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:52.083857 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" Nov 01 00:48:52.084090 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:52.083881 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx" event={"ID":"ae290517-e58a-48ff-8e28-495f3ce917e3","Type":"ContainerDied","Data":"2c44fcc7b1ed1c94e3b28415fd62d591c082ec321eb23929bc578a438542d8ad"} Nov 01 00:48:52.084090 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:48:52.083955 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c44fcc7b1ed1c94e3b28415fd62d591c082ec321eb23929bc578a438542d8ad" Nov 01 00:49:09.063688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:09.063650 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp"] Nov 01 00:49:09.110527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:09.110495 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e630b1a-cl6zp"] Nov 01 00:49:10.194998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:10.194960 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d316eaf-900e-48d1-8a6d-5b35d81bb1b3" path="/var/lib/kubelet/pods/1d316eaf-900e-48d1-8a6d-5b35d81bb1b3/volumes" Nov 01 00:49:10.955941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:10.955839 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2"] Nov 01 00:49:10.956503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:10.956489 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ae290517-e58a-48ff-8e28-495f3ce917e3" containerName="revoke-keys" Nov 01 00:49:10.956550 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:10.956504 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae290517-e58a-48ff-8e28-495f3ce917e3" containerName="revoke-keys" Nov 01 00:49:10.956651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:10.956641 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae290517-e58a-48ff-8e28-495f3ce917e3" containerName="revoke-keys" Nov 01 00:49:10.961556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:10.961533 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.018175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.018149 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2"] Nov 01 00:49:11.044957 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.044927 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.044957 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.044957 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwc4\" (UniqueName: \"kubernetes.io/projected/2f4e1361-e16e-4d9b-8f77-90ac550a6909-kube-api-access-9lwc4\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.045178 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.044983 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.061983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.061954 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:49:11.062101 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.061999 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:49:11.145914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.145880 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.146085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.145916 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwc4\" (UniqueName: \"kubernetes.io/projected/2f4e1361-e16e-4d9b-8f77-90ac550a6909-kube-api-access-9lwc4\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.146085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.146041 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.146637 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.146600 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.148300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.148284 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.207364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.207298 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwc4\" (UniqueName: \"kubernetes.io/projected/2f4e1361-e16e-4d9b-8f77-90ac550a6909-kube-api-access-9lwc4\") pod \"maas-api-revoke-sub-5e4cd57a-gdll2\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.275194 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.275166 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:11.662932 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:11.662903 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2"] Nov 01 00:49:11.663288 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:11.663264 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4e1361_e16e_4d9b_8f77_90ac550a6909.slice/crio-0b6292de57ef07cf85573c8624c939e1746ab3822149e3f95e8507e33b7f6178 WatchSource:0}: Error finding container 0b6292de57ef07cf85573c8624c939e1746ab3822149e3f95e8507e33b7f6178: Status 404 returned error can't find the container with id 0b6292de57ef07cf85573c8624c939e1746ab3822149e3f95e8507e33b7f6178 Nov 01 00:49:12.302935 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:12.302896 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" event={"ID":"2f4e1361-e16e-4d9b-8f77-90ac550a6909","Type":"ContainerStarted","Data":"6460655a0bbf2f8ae798c7ae972a45c43ac4f799bd67b9ad84e30f4aed472df5"} Nov 01 00:49:12.303447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:12.302955 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" event={"ID":"2f4e1361-e16e-4d9b-8f77-90ac550a6909","Type":"ContainerStarted","Data":"0b6292de57ef07cf85573c8624c939e1746ab3822149e3f95e8507e33b7f6178"} Nov 01 00:49:12.421414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:12.421362 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" podStartSLOduration=2.421346868 podStartE2EDuration="2.421346868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:12.418598711 +0000 UTC m=+1542.398181047" watchObservedRunningTime="2026-11-01 00:49:12.421346868 +0000 UTC m=+1542.400929257" Nov 01 00:49:13.103614 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:49:13.103574 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:49:13.313093 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.313058 4910 generic.go:356] "Generic (PLEG): container finished" podID="2f4e1361-e16e-4d9b-8f77-90ac550a6909" containerID="6460655a0bbf2f8ae798c7ae972a45c43ac4f799bd67b9ad84e30f4aed472df5" exitCode=0 Nov 01 00:49:13.313484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.313148 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" event={"ID":"2f4e1361-e16e-4d9b-8f77-90ac550a6909","Type":"ContainerDied","Data":"6460655a0bbf2f8ae798c7ae972a45c43ac4f799bd67b9ad84e30f4aed472df5"} Nov 01 00:49:13.316064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.316040 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk"] Nov 01 00:49:13.320078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.320060 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.415513 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.415437 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-22d7b2\"/\"distinct-2-w6-22d7b2-kserve-self-signed-certs\"" Nov 01 00:49:13.420083 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.420063 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk"] Nov 01 00:49:13.475459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.475424 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-model-cache\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.475700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.475508 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-home\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.475700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.475534 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-tmp-dir\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.475842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.475708 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-dshm\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.475842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.475730 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqfg\" (UniqueName: \"kubernetes.io/projected/58e41946-7966-4980-8379-055ec7181838-kube-api-access-shqfg\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.475842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.475759 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e41946-7966-4980-8379-055ec7181838-tls-certs\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.576460 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.576429 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-dshm\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.576460 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.576464 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqfg\" (UniqueName: \"kubernetes.io/projected/58e41946-7966-4980-8379-055ec7181838-kube-api-access-shqfg\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.576764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.576495 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e41946-7966-4980-8379-055ec7181838-tls-certs\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.576764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.576575 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-model-cache\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.576764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.576678 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-home\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.576764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.576709 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-tmp-dir\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.576948 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.576935 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-model-cache\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.577096 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.577072 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-home\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.577214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.577141 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-tmp-dir\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.578950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.578926 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-dshm\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.579422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.579401 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e41946-7966-4980-8379-055ec7181838-tls-certs\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.639450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.639422 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqfg\" (UniqueName: \"kubernetes.io/projected/58e41946-7966-4980-8379-055ec7181838-kube-api-access-shqfg\") pod \"distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:13.932488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:13.932455 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:14.324440 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:14.324394 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e41946_7966_4980_8379_055ec7181838.slice/crio-9160c3cc2efbc4ef95d4d30b9b6168cf7b903b2696b271c6e3695f3698244238 WatchSource:0}: Error finding container 9160c3cc2efbc4ef95d4d30b9b6168cf7b903b2696b271c6e3695f3698244238: Status 404 returned error can't find the container with id 9160c3cc2efbc4ef95d4d30b9b6168cf7b903b2696b271c6e3695f3698244238 Nov 01 00:49:14.325052 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.325032 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk"] Nov 01 00:49:14.478664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.478609 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:14.593551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.593481 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-xks-ca\") pod \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " Nov 01 00:49:14.593800 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.593684 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4e1361-e16e-4d9b-8f77-90ac550a6909-kube-api-access-9lwc4\" (UniqueName: \"kubernetes.io/projected/2f4e1361-e16e-4d9b-8f77-90ac550a6909-kube-api-access-9lwc4\") pod \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " Nov 01 00:49:14.593800 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.593786 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-openshift-ca\") pod \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\" (UID: \"2f4e1361-e16e-4d9b-8f77-90ac550a6909\") " Nov 01 00:49:14.594210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.594181 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-openshift-ca" pod "2f4e1361-e16e-4d9b-8f77-90ac550a6909" (UID: "2f4e1361-e16e-4d9b-8f77-90ac550a6909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:49:14.596150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.596099 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-xks-ca" pod "2f4e1361-e16e-4d9b-8f77-90ac550a6909" (UID: "2f4e1361-e16e-4d9b-8f77-90ac550a6909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:49:14.597746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.597670 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4e1361-e16e-4d9b-8f77-90ac550a6909-kube-api-access-9lwc4" pod "2f4e1361-e16e-4d9b-8f77-90ac550a6909" (UID: "2f4e1361-e16e-4d9b-8f77-90ac550a6909"). InnerVolumeSpecName "kube-api-access-9lwc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:49:14.695269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.695235 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:14.695269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.695265 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4e1361-e16e-4d9b-8f77-90ac550a6909-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:14.695269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:14.695275 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwc4\" (UniqueName: \"kubernetes.io/projected/2f4e1361-e16e-4d9b-8f77-90ac550a6909-kube-api-access-9lwc4\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:15.338744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:15.338711 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" Nov 01 00:49:15.338744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:15.338739 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2" event={"ID":"2f4e1361-e16e-4d9b-8f77-90ac550a6909","Type":"ContainerDied","Data":"0b6292de57ef07cf85573c8624c939e1746ab3822149e3f95e8507e33b7f6178"} Nov 01 00:49:15.339269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:15.338814 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b6292de57ef07cf85573c8624c939e1746ab3822149e3f95e8507e33b7f6178" Nov 01 00:49:15.340412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:15.340386 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" event={"ID":"58e41946-7966-4980-8379-055ec7181838","Type":"ContainerStarted","Data":"5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1"} Nov 01 00:49:15.340519 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:15.340421 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" event={"ID":"58e41946-7966-4980-8379-055ec7181838","Type":"ContainerStarted","Data":"9160c3cc2efbc4ef95d4d30b9b6168cf7b903b2696b271c6e3695f3698244238"} Nov 01 00:49:15.933116 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:15.933087 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:19.049643 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:19.049576 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podStartSLOduration=6.049561615 podStartE2EDuration="6.049561615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:15.507283549 +0000 UTC m=+1545.486865884" watchObservedRunningTime="2026-11-01 00:49:19.049561615 +0000 UTC m=+1549.029143975" Nov 01 00:49:19.050735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:19.050716 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74"] Nov 01 00:49:19.098497 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:19.098471 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e5cf46-2jg74"] Nov 01 00:49:20.194770 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:20.194731 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174b1886-1f98-47a8-8463-b7f3539be035" path="/var/lib/kubelet/pods/174b1886-1f98-47a8-8463-b7f3539be035/volumes" Nov 01 00:49:22.037714 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.037687 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5"] Nov 01 00:49:22.038285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.038270 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2f4e1361-e16e-4d9b-8f77-90ac550a6909" containerName="revoke-keys" Nov 01 00:49:22.038342 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.038286 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4e1361-e16e-4d9b-8f77-90ac550a6909" containerName="revoke-keys" Nov 01 00:49:22.038386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.038378 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4e1361-e16e-4d9b-8f77-90ac550a6909" containerName="revoke-keys" Nov 01 00:49:22.041976 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.041961 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.149354 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.149322 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5"] Nov 01 00:49:22.186012 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.185978 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186023 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2pn\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-kube-api-access-mz2pn\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186224 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186199 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186234 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186271 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186369 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186436 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186472 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.186522 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.186494 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287495 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287464 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287502 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287526 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287681 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287736 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287787 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287820 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287879 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.287992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.287981 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.288182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.288010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2pn\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-kube-api-access-mz2pn\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.288182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.288113 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.288182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.288130 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.288316 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.288181 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.288603 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.288581 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.289882 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.289864 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.290511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.290487 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.348872 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.348837 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.349015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.348895 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2pn\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-kube-api-access-mz2pn\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.355917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.355885 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:22.808657 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:22.808607 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098fef6a_5bd4_4bda_a5fe_d8d5af5e52ce.slice/crio-302e83309be1022b190f28ff70a438734d16d5a163fc84b145cdbf6cef17d8cc WatchSource:0}: Error finding container 302e83309be1022b190f28ff70a438734d16d5a163fc84b145cdbf6cef17d8cc: Status 404 returned error can't find the container with id 302e83309be1022b190f28ff70a438734d16d5a163fc84b145cdbf6cef17d8cc Nov 01 00:49:22.808821 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.808799 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5"] Nov 01 00:49:22.811210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.811175 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:22.811289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.811245 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:22.811289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:22.811276 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:23.417682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:23.417640 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" event={"ID":"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce","Type":"ContainerStarted","Data":"4848bab028847c1cd5e9a55a1cdf68f1754abd2d3565a24353f906d940b0e13e"} Nov 01 00:49:23.418113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:23.417699 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" event={"ID":"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce","Type":"ContainerStarted","Data":"302e83309be1022b190f28ff70a438734d16d5a163fc84b145cdbf6cef17d8cc"} Nov 01 00:49:23.933682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:23.933650 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:23.946672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:23.946647 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:24.080647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:24.071841 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podStartSLOduration=3.071820781 podStartE2EDuration="3.071820781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:23.582643163 +0000 UTC m=+1553.562225492" watchObservedRunningTime="2026-11-01 00:49:24.071820781 +0000 UTC m=+1554.051403120" Nov 01 00:49:24.358439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:24.358406 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:24.358804 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:24.358781 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 01 00:49:24.358954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:24.358929 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 01 00:49:24.437410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:24.437366 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:49:25.357646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:25.356939 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 01 00:49:25.357646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:25.356995 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 01 00:49:26.085721 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:26.084880 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h"] Nov 01 00:49:26.101813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:26.101783 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde5bcb8-tjj2h"] Nov 01 00:49:26.194064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:26.194034 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101d3974-26c6-49f4-9884-374f485be157" path="/var/lib/kubelet/pods/101d3974-26c6-49f4-9884-374f485be157/volumes" Nov 01 00:49:26.365704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:26.362961 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 01 00:49:26.365704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:26.363028 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 01 00:49:27.337868 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.337832 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km"] Nov 01 00:49:27.342288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.342267 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.367650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.363642 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 01 00:49:27.367650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.363693 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 01 00:49:27.395385 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.395355 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:49:27.445405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.445381 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:49:27.483734 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.483705 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km"] Nov 01 00:49:27.491945 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.491920 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67gf\" (UniqueName: \"kubernetes.io/projected/60d229ca-8be4-48ed-b6dd-e48e327a922c-kube-api-access-c67gf\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.492137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.492110 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.492282 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.492266 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.593616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.593526 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67gf\" (UniqueName: \"kubernetes.io/projected/60d229ca-8be4-48ed-b6dd-e48e327a922c-kube-api-access-c67gf\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.593616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.593595 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.593897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.593692 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.594525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.594426 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.596209 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.596183 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.691394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.670774 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67gf\" (UniqueName: \"kubernetes.io/projected/60d229ca-8be4-48ed-b6dd-e48e327a922c-kube-api-access-c67gf\") pod \"maas-api-revoke-sub-112865ae-227km\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:27.982093 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:27.982001 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:28.360991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:28.360955 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 01 00:49:28.361609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:28.361576 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 01 00:49:28.390202 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:28.390154 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d229ca_8be4_48ed_b6dd_e48e327a922c.slice/crio-dfd34ba6c851a14c3d4ecf844f04fe48c698f70d4644f00f075c783f53a2ce1b WatchSource:0}: Error finding container dfd34ba6c851a14c3d4ecf844f04fe48c698f70d4644f00f075c783f53a2ce1b: Status 404 returned error can't find the container with id dfd34ba6c851a14c3d4ecf844f04fe48c698f70d4644f00f075c783f53a2ce1b Nov 01 00:49:28.392962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:28.392923 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km"] Nov 01 00:49:28.516663 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:28.514121 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" event={"ID":"60d229ca-8be4-48ed-b6dd-e48e327a922c","Type":"ContainerStarted","Data":"dfd34ba6c851a14c3d4ecf844f04fe48c698f70d4644f00f075c783f53a2ce1b"} Nov 01 00:49:29.356814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:29.356778 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 01 00:49:29.356953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:29.356838 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 01 00:49:29.523745 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:29.523708 4910 generic.go:356] "Generic (PLEG): container finished" podID="60d229ca-8be4-48ed-b6dd-e48e327a922c" containerID="28d27392c3880589a546dce052b525f83592cc5eb4943bb51d6817e014e813af" exitCode=0 Nov 01 00:49:29.524193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:29.523804 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" event={"ID":"60d229ca-8be4-48ed-b6dd-e48e327a922c","Type":"ContainerDied","Data":"28d27392c3880589a546dce052b525f83592cc5eb4943bb51d6817e014e813af"} Nov 01 00:49:30.361615 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.361580 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:30.362135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.362108 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:30.363065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.363043 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:49:30.705802 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.705773 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:30.840835 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.840808 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-openshift-ca\") pod \"60d229ca-8be4-48ed-b6dd-e48e327a922c\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " Nov 01 00:49:30.841025 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.840900 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-xks-ca\") pod \"60d229ca-8be4-48ed-b6dd-e48e327a922c\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " Nov 01 00:49:30.841025 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.840955 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d229ca-8be4-48ed-b6dd-e48e327a922c-kube-api-access-c67gf\" (UniqueName: \"kubernetes.io/projected/60d229ca-8be4-48ed-b6dd-e48e327a922c-kube-api-access-c67gf\") pod \"60d229ca-8be4-48ed-b6dd-e48e327a922c\" (UID: \"60d229ca-8be4-48ed-b6dd-e48e327a922c\") " Nov 01 00:49:30.841242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.841217 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-openshift-ca" pod "60d229ca-8be4-48ed-b6dd-e48e327a922c" (UID: "60d229ca-8be4-48ed-b6dd-e48e327a922c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:49:30.841446 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.841425 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:30.843670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.843611 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d229ca-8be4-48ed-b6dd-e48e327a922c-kube-api-access-c67gf" pod "60d229ca-8be4-48ed-b6dd-e48e327a922c" (UID: "60d229ca-8be4-48ed-b6dd-e48e327a922c"). InnerVolumeSpecName "kube-api-access-c67gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:49:30.845088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.845064 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-xks-ca" pod "60d229ca-8be4-48ed-b6dd-e48e327a922c" (UID: "60d229ca-8be4-48ed-b6dd-e48e327a922c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:49:30.942260 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.942183 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c67gf\" (UniqueName: \"kubernetes.io/projected/60d229ca-8be4-48ed-b6dd-e48e327a922c-kube-api-access-c67gf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:30.942260 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:30.942212 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d229ca-8be4-48ed-b6dd-e48e327a922c-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:31.540422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:31.540390 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" Nov 01 00:49:31.540422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:31.540417 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km" event={"ID":"60d229ca-8be4-48ed-b6dd-e48e327a922c","Type":"ContainerDied","Data":"dfd34ba6c851a14c3d4ecf844f04fe48c698f70d4644f00f075c783f53a2ce1b"} Nov 01 00:49:31.540658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:31.540465 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfd34ba6c851a14c3d4ecf844f04fe48c698f70d4644f00f075c783f53a2ce1b" Nov 01 00:49:34.082917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.082873 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n"] Nov 01 00:49:34.132656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.132601 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f732c126-js62n"] Nov 01 00:49:34.218767 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.218729 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb7225d-ef79-4657-85a5-674f4acfeb87" path="/var/lib/kubelet/pods/1eb7225d-ef79-4657-85a5-674f4acfeb87/volumes" Nov 01 00:49:34.614900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.614862 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv"] Nov 01 00:49:34.616087 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.616061 4910 cpu_manager.go:414] "containerMap: removing container" podUID="60d229ca-8be4-48ed-b6dd-e48e327a922c" containerName="revoke-keys" Nov 01 00:49:34.616239 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.616224 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d229ca-8be4-48ed-b6dd-e48e327a922c" containerName="revoke-keys" Nov 01 00:49:34.616540 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.616523 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d229ca-8be4-48ed-b6dd-e48e327a922c" containerName="revoke-keys" Nov 01 00:49:34.622539 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.622519 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.672127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.672097 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv"] Nov 01 00:49:34.677612 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.677589 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:49:34.677859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.677846 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:49:34.688165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.688140 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.688543 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.688474 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2g5\" (UniqueName: \"kubernetes.io/projected/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-kube-api-access-2q2g5\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.688812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.688791 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.865705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.790072 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2g5\" (UniqueName: \"kubernetes.io/projected/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-kube-api-access-2q2g5\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.865705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.790538 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.865705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.791469 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.865705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.792351 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.899648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.895486 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2g5\" (UniqueName: \"kubernetes.io/projected/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-kube-api-access-2q2g5\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.982676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.982033 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-724a1cfd-s8lgv\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:34.982676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:34.982260 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:35.408705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:35.406318 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:49:35.897925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:35.897896 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv"] Nov 01 00:49:36.212819 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.211785 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:49:36.534738 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.534707 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm"] Nov 01 00:49:36.558801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.558768 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm"] Nov 01 00:49:36.559131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.559114 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.645186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.645143 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.645505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.645482 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.645748 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.645705 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.645919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.645900 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmlmd\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-kube-api-access-xmlmd\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.646084 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.646066 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.646368 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.646348 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.646558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.646529 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.649410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.649392 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.649672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.649653 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.762709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.762669 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.762900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.762741 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.762900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.762769 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.762900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.762802 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.762900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.762870 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.763150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.762941 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.763150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.763004 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.763150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.763038 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmlmd\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-kube-api-access-xmlmd\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.763150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.763081 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.763552 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.763523 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.764749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.764721 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.768433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.765220 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.768600 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.765409 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.769016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.768997 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.771917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.771897 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.773789 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.773746 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.838001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.837912 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmlmd\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-kube-api-access-xmlmd\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.838515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.838489 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:36.949369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.949287 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" event={"ID":"3ff287dd-b20c-4bb7-adf1-37be58be6c7a","Type":"ContainerStarted","Data":"c4dac0b32ece0f21db49f8e68de20a73a6f1af943bd2500083fe8efa5142dea3"} Nov 01 00:49:36.967928 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:36.967899 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:37.441696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.441664 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm"] Nov 01 00:49:37.442092 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:37.442065 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508f14e8_3bef_46c0_a6cc_55a430bb9a48.slice/crio-8aadf25487a127d5753c74a81eb60a6836c4c153028ee723930ccd4656a273b8 WatchSource:0}: Error finding container 8aadf25487a127d5753c74a81eb60a6836c4c153028ee723930ccd4656a273b8: Status 404 returned error can't find the container with id 8aadf25487a127d5753c74a81eb60a6836c4c153028ee723930ccd4656a273b8 Nov 01 00:49:37.444868 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.444836 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:37.444955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.444901 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:37.444994 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.444961 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:37.962989 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.962956 4910 generic.go:356] "Generic (PLEG): container finished" podID="3ff287dd-b20c-4bb7-adf1-37be58be6c7a" containerID="32102761f02daaabec9f500fa5071a19ab6cd42d68af997fa5edda2eda7afc33" exitCode=0 Nov 01 00:49:37.963474 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.962989 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" event={"ID":"3ff287dd-b20c-4bb7-adf1-37be58be6c7a","Type":"ContainerDied","Data":"32102761f02daaabec9f500fa5071a19ab6cd42d68af997fa5edda2eda7afc33"} Nov 01 00:49:37.966142 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.966116 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" event={"ID":"508f14e8-3bef-46c0-a6cc-55a430bb9a48","Type":"ContainerStarted","Data":"0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc"} Nov 01 00:49:37.966268 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.966165 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" event={"ID":"508f14e8-3bef-46c0-a6cc-55a430bb9a48","Type":"ContainerStarted","Data":"8aadf25487a127d5753c74a81eb60a6836c4c153028ee723930ccd4656a273b8"} Nov 01 00:49:37.970719 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:37.970693 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:38.211849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:38.211785 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podStartSLOduration=2.211766889 podStartE2EDuration="2.211766889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:38.20842235 +0000 UTC m=+1568.188004682" watchObservedRunningTime="2026-11-01 00:49:38.211766889 +0000 UTC m=+1568.191349225" Nov 01 00:49:38.977715 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:38.976446 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 00:49:38.977715 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:38.976515 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 00:49:39.219138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.219115 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:39.317934 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.317901 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-kube-api-access-2q2g5\" (UniqueName: \"kubernetes.io/projected/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-kube-api-access-2q2g5\") pod \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " Nov 01 00:49:39.318137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.318021 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-xks-ca\") pod \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " Nov 01 00:49:39.318137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.318059 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-openshift-ca\") pod \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\" (UID: \"3ff287dd-b20c-4bb7-adf1-37be58be6c7a\") " Nov 01 00:49:39.318500 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.318467 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-openshift-ca" pod "3ff287dd-b20c-4bb7-adf1-37be58be6c7a" (UID: "3ff287dd-b20c-4bb7-adf1-37be58be6c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:49:39.320171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.320150 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-kube-api-access-2q2g5" pod "3ff287dd-b20c-4bb7-adf1-37be58be6c7a" (UID: "3ff287dd-b20c-4bb7-adf1-37be58be6c7a"). InnerVolumeSpecName "kube-api-access-2q2g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:49:39.320253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.320210 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-xks-ca" pod "3ff287dd-b20c-4bb7-adf1-37be58be6c7a" (UID: "3ff287dd-b20c-4bb7-adf1-37be58be6c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:49:39.419542 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.419512 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2g5\" (UniqueName: \"kubernetes.io/projected/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-kube-api-access-2q2g5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:39.419542 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.419541 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:39.419777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.419554 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff287dd-b20c-4bb7-adf1-37be58be6c7a-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:39.987770 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.978750 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 00:49:39.987770 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:39.978805 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 00:49:40.011711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:40.009461 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" Nov 01 00:49:40.011711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:40.009479 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv" event={"ID":"3ff287dd-b20c-4bb7-adf1-37be58be6c7a","Type":"ContainerDied","Data":"c4dac0b32ece0f21db49f8e68de20a73a6f1af943bd2500083fe8efa5142dea3"} Nov 01 00:49:40.011711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:40.009556 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4dac0b32ece0f21db49f8e68de20a73a6f1af943bd2500083fe8efa5142dea3" Nov 01 00:49:40.977656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:40.976017 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 00:49:40.977656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:40.976078 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 00:49:41.971650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:41.970924 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 00:49:41.971650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:41.970983 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 00:49:42.982650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:42.973998 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 00:49:42.982650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:42.974080 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 00:49:43.067649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:43.066997 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs"] Nov 01 00:49:43.090701 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:43.090664 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b33c5-dlgxs"] Nov 01 00:49:43.115385 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:49:43.115353 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:49:43.969103 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:43.969064 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 01 00:49:43.969310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:43.969127 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 01 00:49:44.196006 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:44.195974 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5e9fc1-f214-415a-8eeb-fc1954933214" path="/var/lib/kubelet/pods/db5e9fc1-f214-415a-8eeb-fc1954933214/volumes" Nov 01 00:49:44.974143 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:44.974111 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:44.974498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:44.974477 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:44.975240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:44.975215 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 00:49:45.082193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:45.082160 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn"] Nov 01 00:49:45.086449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:45.086413 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf8a334-8s6kn"] Nov 01 00:49:46.192839 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:46.192808 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d40c94a-022e-4346-8b9a-d6eb735daa2a" path="/var/lib/kubelet/pods/8d40c94a-022e-4346-8b9a-d6eb735daa2a/volumes" Nov 01 00:49:46.981033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:46.980995 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76"] Nov 01 00:49:46.981686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:46.981666 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3ff287dd-b20c-4bb7-adf1-37be58be6c7a" containerName="revoke-keys" Nov 01 00:49:46.981686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:46.981686 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff287dd-b20c-4bb7-adf1-37be58be6c7a" containerName="revoke-keys" Nov 01 00:49:46.981890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:46.981874 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff287dd-b20c-4bb7-adf1-37be58be6c7a" containerName="revoke-keys" Nov 01 00:49:46.986287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:46.986267 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.089886 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.089840 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:49:47.090133 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.089855 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:49:47.097688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.097659 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76"] Nov 01 00:49:47.152730 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.152689 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.152874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.152754 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v972\" (UniqueName: \"kubernetes.io/projected/594d72da-b160-4647-a3c6-8af7aae13eb2-kube-api-access-7v972\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.152924 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.152900 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.253694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.253663 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.254166 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.253798 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.254166 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.253841 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v972\" (UniqueName: \"kubernetes.io/projected/594d72da-b160-4647-a3c6-8af7aae13eb2-kube-api-access-7v972\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.254357 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.254337 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.256356 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.256336 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.316232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.316204 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v972\" (UniqueName: \"kubernetes.io/projected/594d72da-b160-4647-a3c6-8af7aae13eb2-kube-api-access-7v972\") pod \"maas-api-revoke-sub-91473b4d-97w76\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.601442 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.601362 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:47.957956 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.957867 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj"] Nov 01 00:49:47.963160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.963136 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:47.998012 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:47.995951 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76"] Nov 01 00:49:48.000279 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:48.000239 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594d72da_b160_4647_a3c6_8af7aae13eb2.slice/crio-d811547a171666a3edee08cdba31c45700b833cd963dd87c074bd4fb0887b449 WatchSource:0}: Error finding container d811547a171666a3edee08cdba31c45700b833cd963dd87c074bd4fb0887b449: Status 404 returned error can't find the container with id d811547a171666a3edee08cdba31c45700b833cd963dd87c074bd4fb0887b449 Nov 01 00:49:48.020257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.020228 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj"] Nov 01 00:49:48.063835 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.063804 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.064024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.064006 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.064244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.064220 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j878h\" (UniqueName: \"kubernetes.io/projected/e707298e-d9a3-4d29-96f1-b15b5759a9d1-kube-api-access-j878h\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.074013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.073987 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" event={"ID":"594d72da-b160-4647-a3c6-8af7aae13eb2","Type":"ContainerStarted","Data":"d811547a171666a3edee08cdba31c45700b833cd963dd87c074bd4fb0887b449"} Nov 01 00:49:48.165263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.165234 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j878h\" (UniqueName: \"kubernetes.io/projected/e707298e-d9a3-4d29-96f1-b15b5759a9d1-kube-api-access-j878h\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.165384 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.165293 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.165438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.165417 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.166010 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.165986 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.167981 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.167962 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.226362 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.226297 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j878h\" (UniqueName: \"kubernetes.io/projected/e707298e-d9a3-4d29-96f1-b15b5759a9d1-kube-api-access-j878h\") pod \"maas-api-revoke-sub-1821359e-9r8sj\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.281824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.281803 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:48.667915 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:48.667882 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode707298e_d9a3_4d29_96f1_b15b5759a9d1.slice/crio-a549d7948d9c4b93e17d8c8da3ef7bec4e355ad2aed16a0eb8f54873abd3abdc WatchSource:0}: Error finding container a549d7948d9c4b93e17d8c8da3ef7bec4e355ad2aed16a0eb8f54873abd3abdc: Status 404 returned error can't find the container with id a549d7948d9c4b93e17d8c8da3ef7bec4e355ad2aed16a0eb8f54873abd3abdc Nov 01 00:49:48.669333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:48.669305 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj"] Nov 01 00:49:49.084159 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:49.084130 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" event={"ID":"e707298e-d9a3-4d29-96f1-b15b5759a9d1","Type":"ContainerStarted","Data":"3e648724823f5140a86d23debfc956adcd47299a96870b95f66e896b437615f1"} Nov 01 00:49:49.084300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:49.084190 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" event={"ID":"e707298e-d9a3-4d29-96f1-b15b5759a9d1","Type":"ContainerStarted","Data":"a549d7948d9c4b93e17d8c8da3ef7bec4e355ad2aed16a0eb8f54873abd3abdc"} Nov 01 00:49:49.085763 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:49.085737 4910 generic.go:356] "Generic (PLEG): container finished" podID="594d72da-b160-4647-a3c6-8af7aae13eb2" containerID="dd45525d28d06ba2db3d4da4b5c419d58ec2e867a9acfff392aba5a1000a6c0f" exitCode=0 Nov 01 00:49:49.085862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:49.085802 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" event={"ID":"594d72da-b160-4647-a3c6-8af7aae13eb2","Type":"ContainerDied","Data":"dd45525d28d06ba2db3d4da4b5c419d58ec2e867a9acfff392aba5a1000a6c0f"} Nov 01 00:49:49.206333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:49.206244 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" podStartSLOduration=2.206231281 podStartE2EDuration="2.206231281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:49.201249568 +0000 UTC m=+1579.180831906" watchObservedRunningTime="2026-11-01 00:49:49.206231281 +0000 UTC m=+1579.185813615" Nov 01 00:49:50.095322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.095292 4910 generic.go:356] "Generic (PLEG): container finished" podID="e707298e-d9a3-4d29-96f1-b15b5759a9d1" containerID="3e648724823f5140a86d23debfc956adcd47299a96870b95f66e896b437615f1" exitCode=0 Nov 01 00:49:50.095760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.095382 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" event={"ID":"e707298e-d9a3-4d29-96f1-b15b5759a9d1","Type":"ContainerDied","Data":"3e648724823f5140a86d23debfc956adcd47299a96870b95f66e896b437615f1"} Nov 01 00:49:50.268049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.268017 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:50.391516 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.391444 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594d72da-b160-4647-a3c6-8af7aae13eb2-kube-api-access-7v972\" (UniqueName: \"kubernetes.io/projected/594d72da-b160-4647-a3c6-8af7aae13eb2-kube-api-access-7v972\") pod \"594d72da-b160-4647-a3c6-8af7aae13eb2\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " Nov 01 00:49:50.391692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.391548 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-xks-ca\") pod \"594d72da-b160-4647-a3c6-8af7aae13eb2\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " Nov 01 00:49:50.391749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.391715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-openshift-ca\") pod \"594d72da-b160-4647-a3c6-8af7aae13eb2\" (UID: \"594d72da-b160-4647-a3c6-8af7aae13eb2\") " Nov 01 00:49:50.392116 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.392086 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-openshift-ca" pod "594d72da-b160-4647-a3c6-8af7aae13eb2" (UID: "594d72da-b160-4647-a3c6-8af7aae13eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:49:50.392277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.392191 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:50.393911 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.393880 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-xks-ca" pod "594d72da-b160-4647-a3c6-8af7aae13eb2" (UID: "594d72da-b160-4647-a3c6-8af7aae13eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:49:50.394081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.394062 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594d72da-b160-4647-a3c6-8af7aae13eb2-kube-api-access-7v972" pod "594d72da-b160-4647-a3c6-8af7aae13eb2" (UID: "594d72da-b160-4647-a3c6-8af7aae13eb2"). InnerVolumeSpecName "kube-api-access-7v972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:49:50.493291 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.493255 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v972\" (UniqueName: \"kubernetes.io/projected/594d72da-b160-4647-a3c6-8af7aae13eb2-kube-api-access-7v972\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:50.493291 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:50.493286 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594d72da-b160-4647-a3c6-8af7aae13eb2-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:51.104596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.104564 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" Nov 01 00:49:51.105073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.104600 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76" event={"ID":"594d72da-b160-4647-a3c6-8af7aae13eb2","Type":"ContainerDied","Data":"d811547a171666a3edee08cdba31c45700b833cd963dd87c074bd4fb0887b449"} Nov 01 00:49:51.105073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.104701 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d811547a171666a3edee08cdba31c45700b833cd963dd87c074bd4fb0887b449" Nov 01 00:49:51.267417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.267388 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:51.402842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.402750 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-openshift-ca\") pod \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " Nov 01 00:49:51.403011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.402973 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-xks-ca\") pod \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " Nov 01 00:49:51.403086 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.403071 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e707298e-d9a3-4d29-96f1-b15b5759a9d1-kube-api-access-j878h\" (UniqueName: \"kubernetes.io/projected/e707298e-d9a3-4d29-96f1-b15b5759a9d1-kube-api-access-j878h\") pod \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\" (UID: \"e707298e-d9a3-4d29-96f1-b15b5759a9d1\") " Nov 01 00:49:51.403142 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.403115 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-openshift-ca" pod "e707298e-d9a3-4d29-96f1-b15b5759a9d1" (UID: "e707298e-d9a3-4d29-96f1-b15b5759a9d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:49:51.403569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.403550 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:51.406345 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.406318 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-xks-ca" pod "e707298e-d9a3-4d29-96f1-b15b5759a9d1" (UID: "e707298e-d9a3-4d29-96f1-b15b5759a9d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:49:51.406484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.406355 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e707298e-d9a3-4d29-96f1-b15b5759a9d1-kube-api-access-j878h" pod "e707298e-d9a3-4d29-96f1-b15b5759a9d1" (UID: "e707298e-d9a3-4d29-96f1-b15b5759a9d1"). InnerVolumeSpecName "kube-api-access-j878h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:49:51.504265 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.504236 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e707298e-d9a3-4d29-96f1-b15b5759a9d1-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:51.504265 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.504261 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j878h\" (UniqueName: \"kubernetes.io/projected/e707298e-d9a3-4d29-96f1-b15b5759a9d1-kube-api-access-j878h\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:49:51.997542 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.997505 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x"] Nov 01 00:49:51.998388 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.998371 4910 cpu_manager.go:414] "containerMap: removing container" podUID="594d72da-b160-4647-a3c6-8af7aae13eb2" containerName="revoke-keys" Nov 01 00:49:51.998462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.998390 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594d72da-b160-4647-a3c6-8af7aae13eb2" containerName="revoke-keys" Nov 01 00:49:51.998462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.998416 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e707298e-d9a3-4d29-96f1-b15b5759a9d1" containerName="revoke-keys" Nov 01 00:49:51.998462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.998424 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e707298e-d9a3-4d29-96f1-b15b5759a9d1" containerName="revoke-keys" Nov 01 00:49:51.998610 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.998599 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e707298e-d9a3-4d29-96f1-b15b5759a9d1" containerName="revoke-keys" Nov 01 00:49:51.998684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:51.998615 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="594d72da-b160-4647-a3c6-8af7aae13eb2" containerName="revoke-keys" Nov 01 00:49:52.002869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.002850 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.110531 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.110492 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.110605 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.110727 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2972c8-7d7d-4eee-9032-54210747636a-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.110853 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxgk\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-kube-api-access-8hxgk\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.110946 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2972c8-7d7d-4eee-9032-54210747636a-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111211 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.111042 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111211 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.111074 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111211 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.111131 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.111211 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.111176 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.117184 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.117162 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" Nov 01 00:49:52.117308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.117187 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj" event={"ID":"e707298e-d9a3-4d29-96f1-b15b5759a9d1","Type":"ContainerDied","Data":"a549d7948d9c4b93e17d8c8da3ef7bec4e355ad2aed16a0eb8f54873abd3abdc"} Nov 01 00:49:52.117308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.117244 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a549d7948d9c4b93e17d8c8da3ef7bec4e355ad2aed16a0eb8f54873abd3abdc" Nov 01 00:49:52.207527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.207497 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x"] Nov 01 00:49:52.212155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212130 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212314 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212171 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212314 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212221 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2972c8-7d7d-4eee-9032-54210747636a-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212314 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212307 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxgk\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-kube-api-access-8hxgk\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212389 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2972c8-7d7d-4eee-9032-54210747636a-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212424 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212448 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212499 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212528 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212567 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.212876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.212853 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-credential-socket\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.213080 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.213057 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-data\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.213289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.213269 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-certs\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.214604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.214576 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2972c8-7d7d-4eee-9032-54210747636a-istiod-ca-cert\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.217407 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.217138 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2972c8-7d7d-4eee-9032-54210747636a-istio-podinfo\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.218156 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.218137 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-envoy\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.274770 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.274696 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxgk\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-kube-api-access-8hxgk\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.277269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.277247 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-istio-token\") pod \"e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.316766 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.316732 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:52.834727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.834696 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x"] Nov 01 00:49:52.834866 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:52.834770 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2972c8_7d7d_4eee_9032_54210747636a.slice/crio-3cd9b868d9b00999f72f030fdb282a7470143adefd0312ba756a1c22a54e9c0e WatchSource:0}: Error finding container 3cd9b868d9b00999f72f030fdb282a7470143adefd0312ba756a1c22a54e9c0e: Status 404 returned error can't find the container with id 3cd9b868d9b00999f72f030fdb282a7470143adefd0312ba756a1c22a54e9c0e Nov 01 00:49:52.837533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.837493 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:52.837697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.837566 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:52.837697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:52.837608 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:49:53.128232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:53.128142 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" event={"ID":"bd2972c8-7d7d-4eee-9032-54210747636a","Type":"ContainerStarted","Data":"12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672"} Nov 01 00:49:53.128232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:53.128193 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" event={"ID":"bd2972c8-7d7d-4eee-9032-54210747636a","Type":"ContainerStarted","Data":"3cd9b868d9b00999f72f030fdb282a7470143adefd0312ba756a1c22a54e9c0e"} Nov 01 00:49:53.253382 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:53.253325 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podStartSLOduration=2.253310712 podStartE2EDuration="2.253310712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:53.248178931 +0000 UTC m=+1583.227761290" watchObservedRunningTime="2026-11-01 00:49:53.253310712 +0000 UTC m=+1583.232893046" Nov 01 00:49:53.317819 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:53.317787 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:49:53.319370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:53.319341 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 01 00:49:53.319494 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:53.319403 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 01 00:49:54.317988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:54.317952 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 01 00:49:54.318404 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:54.318015 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 01 00:49:55.317216 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:55.317183 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 01 00:49:55.317411 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:55.317255 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 01 00:49:56.076741 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:56.076691 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v"] Nov 01 00:49:56.113973 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:56.113936 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fbebc0-t5q9v"] Nov 01 00:49:56.193403 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:56.193368 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904f3f42-3c45-4490-b25d-996ba6b26bbf" path="/var/lib/kubelet/pods/904f3f42-3c45-4490-b25d-996ba6b26bbf/volumes" Nov 01 00:49:56.317580 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:56.317548 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 01 00:49:56.317774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:56.317603 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 01 00:49:57.317174 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.317144 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 01 00:49:57.317552 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.317198 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 01 00:49:57.737497 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.737415 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb"] Nov 01 00:49:57.742713 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.742691 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.795358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.795331 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:49:57.795613 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.795592 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:49:57.807952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.807917 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.808229 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.808205 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.808352 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.808324 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllfp\" (UniqueName: \"kubernetes.io/projected/f4b04cc1-e01d-48be-a56c-2606e48b6704-kube-api-access-sllfp\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.878704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.875670 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb"] Nov 01 00:49:57.909816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.909786 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.910021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.910007 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.910249 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.910224 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllfp\" (UniqueName: \"kubernetes.io/projected/f4b04cc1-e01d-48be-a56c-2606e48b6704-kube-api-access-sllfp\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.910742 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.910379 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.914122 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.914096 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:57.981648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:57.981403 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllfp\" (UniqueName: \"kubernetes.io/projected/f4b04cc1-e01d-48be-a56c-2606e48b6704-kube-api-access-sllfp\") pod \"maas-api-revoke-sub-526a5487-g6ztb\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:58.068100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:58.067850 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:49:58.317238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:58.317166 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 01 00:49:58.317238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:58.317222 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 01 00:49:58.478188 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:49:58.478134 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b04cc1_e01d_48be_a56c_2606e48b6704.slice/crio-ca83fcc34bf6e10cd92498329341ad4a4aec2ed2344df01b1910e9806f3e7142 WatchSource:0}: Error finding container ca83fcc34bf6e10cd92498329341ad4a4aec2ed2344df01b1910e9806f3e7142: Status 404 returned error can't find the container with id ca83fcc34bf6e10cd92498329341ad4a4aec2ed2344df01b1910e9806f3e7142 Nov 01 00:49:58.479653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:58.479604 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb"] Nov 01 00:49:59.222567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:59.222471 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" event={"ID":"f4b04cc1-e01d-48be-a56c-2606e48b6704","Type":"ContainerStarted","Data":"c97424f858faa00424a5a1a244117d715033d96b1a22b40f922b1fc13401a58d"} Nov 01 00:49:59.222567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:59.222535 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" event={"ID":"f4b04cc1-e01d-48be-a56c-2606e48b6704","Type":"ContainerStarted","Data":"ca83fcc34bf6e10cd92498329341ad4a4aec2ed2344df01b1910e9806f3e7142"} Nov 01 00:49:59.317240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:59.317207 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 01 00:49:59.317755 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:59.317264 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 01 00:49:59.339447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:49:59.339390 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" podStartSLOduration=2.339377446 podStartE2EDuration="2.339377446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:49:59.338096078 +0000 UTC m=+1589.317678413" watchObservedRunningTime="2026-11-01 00:49:59.339377446 +0000 UTC m=+1589.318959780" Nov 01 00:50:00.248781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:00.248744 4910 generic.go:356] "Generic (PLEG): container finished" podID="f4b04cc1-e01d-48be-a56c-2606e48b6704" containerID="c97424f858faa00424a5a1a244117d715033d96b1a22b40f922b1fc13401a58d" exitCode=0 Nov 01 00:50:00.248992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:00.248876 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" event={"ID":"f4b04cc1-e01d-48be-a56c-2606e48b6704","Type":"ContainerDied","Data":"c97424f858faa00424a5a1a244117d715033d96b1a22b40f922b1fc13401a58d"} Nov 01 00:50:00.323273 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:00.323239 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:50:00.323680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:00.323575 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:50:00.324373 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:00.324354 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:50:01.419028 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.419006 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:50:01.455717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.455690 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b04cc1-e01d-48be-a56c-2606e48b6704-kube-api-access-sllfp\" (UniqueName: \"kubernetes.io/projected/f4b04cc1-e01d-48be-a56c-2606e48b6704-kube-api-access-sllfp\") pod \"f4b04cc1-e01d-48be-a56c-2606e48b6704\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " Nov 01 00:50:01.455869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.455825 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-xks-ca\") pod \"f4b04cc1-e01d-48be-a56c-2606e48b6704\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " Nov 01 00:50:01.455951 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.455895 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-openshift-ca\") pod \"f4b04cc1-e01d-48be-a56c-2606e48b6704\" (UID: \"f4b04cc1-e01d-48be-a56c-2606e48b6704\") " Nov 01 00:50:01.456223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.456195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-openshift-ca" pod "f4b04cc1-e01d-48be-a56c-2606e48b6704" (UID: "f4b04cc1-e01d-48be-a56c-2606e48b6704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:01.456686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.456658 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:01.457971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.457949 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b04cc1-e01d-48be-a56c-2606e48b6704-kube-api-access-sllfp" pod "f4b04cc1-e01d-48be-a56c-2606e48b6704" (UID: "f4b04cc1-e01d-48be-a56c-2606e48b6704"). InnerVolumeSpecName "kube-api-access-sllfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:01.458101 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.458085 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-xks-ca" pod "f4b04cc1-e01d-48be-a56c-2606e48b6704" (UID: "f4b04cc1-e01d-48be-a56c-2606e48b6704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:01.558424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.558312 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sllfp\" (UniqueName: \"kubernetes.io/projected/f4b04cc1-e01d-48be-a56c-2606e48b6704-kube-api-access-sllfp\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:01.558424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:01.558379 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4b04cc1-e01d-48be-a56c-2606e48b6704-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:02.269111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.269084 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" Nov 01 00:50:02.269304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.269114 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb" event={"ID":"f4b04cc1-e01d-48be-a56c-2606e48b6704","Type":"ContainerDied","Data":"ca83fcc34bf6e10cd92498329341ad4a4aec2ed2344df01b1910e9806f3e7142"} Nov 01 00:50:02.269304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.269168 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca83fcc34bf6e10cd92498329341ad4a4aec2ed2344df01b1910e9806f3e7142" Nov 01 00:50:02.446320 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.446289 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd"] Nov 01 00:50:02.446963 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.446948 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f4b04cc1-e01d-48be-a56c-2606e48b6704" containerName="revoke-keys" Nov 01 00:50:02.447012 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.446964 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b04cc1-e01d-48be-a56c-2606e48b6704" containerName="revoke-keys" Nov 01 00:50:02.447067 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.447058 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b04cc1-e01d-48be-a56c-2606e48b6704" containerName="revoke-keys" Nov 01 00:50:02.450792 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.450750 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.503772 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.503731 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:50:02.503971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.503888 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:50:02.508222 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.508200 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd"] Nov 01 00:50:02.571606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.571574 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.571852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.571616 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.571852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.571786 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtv5\" (UniqueName: \"kubernetes.io/projected/c9e800fb-e72f-4961-8f97-88baad1a6f4d-kube-api-access-krtv5\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.621463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.621429 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg"] Nov 01 00:50:02.625930 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.625896 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.626094 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.626070 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx"] Nov 01 00:50:02.630859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.630843 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.664635 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.664605 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72"] Nov 01 00:50:02.669853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.669834 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.673286 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673263 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.673369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673300 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.673419 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673396 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtv5\" (UniqueName: \"kubernetes.io/projected/c9e800fb-e72f-4961-8f97-88baad1a6f4d-kube-api-access-krtv5\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.673456 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673419 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwhb\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-kube-api-access-pgwhb\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.673488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673456 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.673558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673543 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.673593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673573 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.673649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673607 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.673749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673722 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.673796 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673768 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.673856 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673841 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.673887 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673877 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.673951 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673937 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.673988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.673975 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.674029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674014 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.674086 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674031 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.674129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674113 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.674163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674141 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.674209 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674171 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.674258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674213 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4wb\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-kube-api-access-8j4wb\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.674258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674242 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.674801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.674781 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.676690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.676671 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.717027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.715100 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg"] Nov 01 00:50:02.728240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.728066 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx"] Nov 01 00:50:02.739424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.739061 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtv5\" (UniqueName: \"kubernetes.io/projected/c9e800fb-e72f-4961-8f97-88baad1a6f4d-kube-api-access-krtv5\") pod \"maas-api-revoke-sub-77141d38-phxrd\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.764604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.764574 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72"] Nov 01 00:50:02.764775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.764650 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:02.775914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.775887 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwhb\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-kube-api-access-pgwhb\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.775937 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776021 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776049 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.776242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776084 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776111 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.776242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776137 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776178 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776202 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776218 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776260 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776278 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776292 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776341 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776357 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776375 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776396 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776413 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530d5660-94c7-4ae6-a7c4-e825d375d84d-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776441 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776486 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776517 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776545 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-kube-api-access-2944m\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776567 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.776599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776599 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4wb\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-kube-api-access-8j4wb\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.777348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776659 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.777348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776689 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.777348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.776709 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.777348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.777340 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.777580 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.777563 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.778103 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.778078 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.779016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.778986 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.779137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.779122 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.779287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.779253 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.779405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.779317 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.779573 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.779551 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.779705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.779656 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.779921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.779904 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.782518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.782495 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.782728 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.782502 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.782815 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.782796 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.784419 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.784399 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.839820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.839771 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.840128 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.840105 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4wb\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-kube-api-access-8j4wb\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.840179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.840132 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.840560 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.840545 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwhb\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-kube-api-access-pgwhb\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879062 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530d5660-94c7-4ae6-a7c4-e825d375d84d-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879125 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879173 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879213 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879245 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-kube-api-access-2944m\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879347 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879388 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879440 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.879561 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.880540 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.881226 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530d5660-94c7-4ae6-a7c4-e825d375d84d-istiod-ca-cert\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.881484 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-credential-socket\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.883680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.881761 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-data\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.884457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.884233 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-envoy\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.892213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.884780 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-certs\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.892213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.891670 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-podinfo\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.943238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.942767 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:02.945794 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.945766 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-kube-api-access-2944m\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.945955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.945937 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-token\") pod \"e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:02.961913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.949748 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:02.996102 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:02.996068 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:03.194435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.194409 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd"] Nov 01 00:50:03.194586 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:03.194562 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e800fb_e72f_4961_8f97_88baad1a6f4d.slice/crio-d189840ba9dcd2e792fdc76e1d3a2791bf99a751f0985b6c4a8037dd38d9ce75 WatchSource:0}: Error finding container d189840ba9dcd2e792fdc76e1d3a2791bf99a751f0985b6c4a8037dd38d9ce75: Status 404 returned error can't find the container with id d189840ba9dcd2e792fdc76e1d3a2791bf99a751f0985b6c4a8037dd38d9ce75 Nov 01 00:50:03.281065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.281030 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" event={"ID":"c9e800fb-e72f-4961-8f97-88baad1a6f4d","Type":"ContainerStarted","Data":"d189840ba9dcd2e792fdc76e1d3a2791bf99a751f0985b6c4a8037dd38d9ce75"} Nov 01 00:50:03.296763 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.296708 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf"] Nov 01 00:50:03.304152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.304126 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.352734 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.352496 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf"] Nov 01 00:50:03.363387 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:03.363357 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5a8676_5f71_4c35_aed8_b373c47c3a9b.slice/crio-a276dffc20aa14aa8015234736166a685e50b6d82c14f5280b82d8544042e7ac WatchSource:0}: Error finding container a276dffc20aa14aa8015234736166a685e50b6d82c14f5280b82d8544042e7ac: Status 404 returned error can't find the container with id a276dffc20aa14aa8015234736166a685e50b6d82c14f5280b82d8544042e7ac Nov 01 00:50:03.363883 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.363854 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg"] Nov 01 00:50:03.367090 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.367048 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.367245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.367132 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.367245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.367175 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.396256 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.396228 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.396375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.396279 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.396375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.396301 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbch2\" (UniqueName: \"kubernetes.io/projected/e72699e9-14fb-4cd4-90ac-676afb83f9c9-kube-api-access-gbch2\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.405812 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:03.405777 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f37726_d65c_4fdc_9f81_1ca081a7b8ab.slice/crio-5993a73a70708067824ff4de47809682002e34e47255d41fa41c92e036ac5396 WatchSource:0}: Error finding container 5993a73a70708067824ff4de47809682002e34e47255d41fa41c92e036ac5396: Status 404 returned error can't find the container with id 5993a73a70708067824ff4de47809682002e34e47255d41fa41c92e036ac5396 Nov 01 00:50:03.407425 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.407394 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx"] Nov 01 00:50:03.408047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.408018 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.408120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.408078 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.408120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.408106 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.457516 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:03.457474 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530d5660_94c7_4ae6_a7c4_e825d375d84d.slice/crio-53b592b8f9b23af444e3ef38d8973b312f68e62b005cd5579577b29871d5666b WatchSource:0}: Error finding container 53b592b8f9b23af444e3ef38d8973b312f68e62b005cd5579577b29871d5666b: Status 404 returned error can't find the container with id 53b592b8f9b23af444e3ef38d8973b312f68e62b005cd5579577b29871d5666b Nov 01 00:50:03.459047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.459026 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72"] Nov 01 00:50:03.460397 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.460363 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.460484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.460432 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.460484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.460477 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:03.498455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.498425 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.498563 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.498480 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.498563 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.498508 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbch2\" (UniqueName: \"kubernetes.io/projected/e72699e9-14fb-4cd4-90ac-676afb83f9c9-kube-api-access-gbch2\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.499337 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.499288 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.501834 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.501814 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.565647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.565327 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbch2\" (UniqueName: \"kubernetes.io/projected/e72699e9-14fb-4cd4-90ac-676afb83f9c9-kube-api-access-gbch2\") pod \"maas-api-revoke-sub-7ebe7138-5t5gf\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:03.619648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:03.619075 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:04.035324 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:04.035293 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72699e9_14fb_4cd4_90ac_676afb83f9c9.slice/crio-dd84802d21809dac64c397168cb0f6ec2d6642a9d498ad622af318ace1133101 WatchSource:0}: Error finding container dd84802d21809dac64c397168cb0f6ec2d6642a9d498ad622af318ace1133101: Status 404 returned error can't find the container with id dd84802d21809dac64c397168cb0f6ec2d6642a9d498ad622af318ace1133101 Nov 01 00:50:04.038751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.038725 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf"] Nov 01 00:50:04.293004 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.292971 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" event={"ID":"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab","Type":"ContainerStarted","Data":"91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7"} Nov 01 00:50:04.293136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.293034 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" event={"ID":"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab","Type":"ContainerStarted","Data":"5993a73a70708067824ff4de47809682002e34e47255d41fa41c92e036ac5396"} Nov 01 00:50:04.295831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.295800 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" event={"ID":"530d5660-94c7-4ae6-a7c4-e825d375d84d","Type":"ContainerStarted","Data":"57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400"} Nov 01 00:50:04.296009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.295995 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" event={"ID":"530d5660-94c7-4ae6-a7c4-e825d375d84d","Type":"ContainerStarted","Data":"53b592b8f9b23af444e3ef38d8973b312f68e62b005cd5579577b29871d5666b"} Nov 01 00:50:04.297888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.297851 4910 generic.go:356] "Generic (PLEG): container finished" podID="c9e800fb-e72f-4961-8f97-88baad1a6f4d" containerID="45a6c293a4c3189339a50f183c1b41320bdad1eb1449f4f580b97789e1dcbd83" exitCode=0 Nov 01 00:50:04.298022 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.297994 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" event={"ID":"c9e800fb-e72f-4961-8f97-88baad1a6f4d","Type":"ContainerDied","Data":"45a6c293a4c3189339a50f183c1b41320bdad1eb1449f4f580b97789e1dcbd83"} Nov 01 00:50:04.302092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.302067 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" event={"ID":"e72699e9-14fb-4cd4-90ac-676afb83f9c9","Type":"ContainerStarted","Data":"dd84802d21809dac64c397168cb0f6ec2d6642a9d498ad622af318ace1133101"} Nov 01 00:50:04.307996 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.307973 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" event={"ID":"2d5a8676-5f71-4c35-aed8-b373c47c3a9b","Type":"ContainerStarted","Data":"0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05"} Nov 01 00:50:04.308104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.308018 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" event={"ID":"2d5a8676-5f71-4c35-aed8-b373c47c3a9b","Type":"ContainerStarted","Data":"a276dffc20aa14aa8015234736166a685e50b6d82c14f5280b82d8544042e7ac"} Nov 01 00:50:04.427478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.427415 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podStartSLOduration=2.427395821 podStartE2EDuration="2.427395821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:04.424803383 +0000 UTC m=+1594.404385719" watchObservedRunningTime="2026-11-01 00:50:04.427395821 +0000 UTC m=+1594.406978157" Nov 01 00:50:04.703751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.697905 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podStartSLOduration=2.697885698 podStartE2EDuration="2.697885698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:04.571014279 +0000 UTC m=+1594.550596616" watchObservedRunningTime="2026-11-01 00:50:04.697885698 +0000 UTC m=+1594.677468035" Nov 01 00:50:04.912649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.910447 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podStartSLOduration=2.910423883 podStartE2EDuration="2.910423883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:04.838004404 +0000 UTC m=+1594.817586740" watchObservedRunningTime="2026-11-01 00:50:04.910423883 +0000 UTC m=+1594.890006219" Nov 01 00:50:04.948670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.945691 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:04.948670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.947188 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:04.948670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.947237 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:04.960784 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:04.960693 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:05.000042 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:05.000007 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:05.000444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:05.000352 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:05.000692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:05.000666 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:05.004756 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:05.004731 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:05.005338 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:05.005304 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:05.355661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:05.354519 4910 generic.go:356] "Generic (PLEG): container finished" podID="e72699e9-14fb-4cd4-90ac-676afb83f9c9" containerID="72de6f4e20ec30a1ec28ee444f45e71cb8695efc433492c1a02187670f026f25" exitCode=0 Nov 01 00:50:05.355661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:05.355325 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" event={"ID":"e72699e9-14fb-4cd4-90ac-676afb83f9c9","Type":"ContainerDied","Data":"72de6f4e20ec30a1ec28ee444f45e71cb8695efc433492c1a02187670f026f25"} Nov 01 00:50:06.248941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.172108 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:06.248941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.172168 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:06.248941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.187024 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:06.248941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.187089 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:06.343706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.249886 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:06.343706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.249955 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:06.517645 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.507725 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz"] Nov 01 00:50:06.547171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.547139 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.548832 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.548756 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:06.558385 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.558357 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz"] Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562023 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e800fb-e72f-4961-8f97-88baad1a6f4d-kube-api-access-krtv5\" (UniqueName: \"kubernetes.io/projected/c9e800fb-e72f-4961-8f97-88baad1a6f4d-kube-api-access-krtv5\") pod \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562091 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-xks-ca\") pod \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562304 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-openshift-ca\") pod \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\" (UID: \"c9e800fb-e72f-4961-8f97-88baad1a6f4d\") " Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562671 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562766 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562825 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562850 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.562985 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld7xr\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-kube-api-access-ld7xr\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.563089 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.563203 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.563308 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.565699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.563512 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.569296 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.569261 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-openshift-ca" pod "c9e800fb-e72f-4961-8f97-88baad1a6f4d" (UID: "c9e800fb-e72f-4961-8f97-88baad1a6f4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:06.638089 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.638011 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e800fb-e72f-4961-8f97-88baad1a6f4d-kube-api-access-krtv5" pod "c9e800fb-e72f-4961-8f97-88baad1a6f4d" (UID: "c9e800fb-e72f-4961-8f97-88baad1a6f4d"). InnerVolumeSpecName "kube-api-access-krtv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:06.646524 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.646475 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-xks-ca" pod "c9e800fb-e72f-4961-8f97-88baad1a6f4d" (UID: "c9e800fb-e72f-4961-8f97-88baad1a6f4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.668578 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.668738 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.668809 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.668869 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.668897 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.668922 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.669010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld7xr\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-kube-api-access-ld7xr\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.669070 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.669133 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.669242 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krtv5\" (UniqueName: \"kubernetes.io/projected/c9e800fb-e72f-4961-8f97-88baad1a6f4d-kube-api-access-krtv5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.669260 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.669277 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e800fb-e72f-4961-8f97-88baad1a6f4d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.669718 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-credential-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.674300 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.670683 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-socket\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.713752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.713243 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-certs\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.732699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.721412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-envoy\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.732699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.722044 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-data\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.732699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.723438 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istiod-ca-cert\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.734198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.734141 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-podinfo\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.804799 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.804717 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld7xr\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-kube-api-access-ld7xr\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.838691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.838654 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-token\") pod \"e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:06.914647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:06.910887 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:07.028180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.027502 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.028180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.027561 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:07.052647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.044273 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.052647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.044341 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:07.154479 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.154317 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.154479 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.154389 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:07.261676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.253716 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:50:07.461758 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.460711 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:07.467507 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.467170 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:07.511654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.511231 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" event={"ID":"c9e800fb-e72f-4961-8f97-88baad1a6f4d","Type":"ContainerDied","Data":"d189840ba9dcd2e792fdc76e1d3a2791bf99a751f0985b6c4a8037dd38d9ce75"} Nov 01 00:50:07.511654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.511296 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d189840ba9dcd2e792fdc76e1d3a2791bf99a751f0985b6c4a8037dd38d9ce75" Nov 01 00:50:07.511654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.511362 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd" Nov 01 00:50:07.525652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.520389 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" event={"ID":"e72699e9-14fb-4cd4-90ac-676afb83f9c9","Type":"ContainerDied","Data":"dd84802d21809dac64c397168cb0f6ec2d6642a9d498ad622af318ace1133101"} Nov 01 00:50:07.525652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.520442 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd84802d21809dac64c397168cb0f6ec2d6642a9d498ad622af318ace1133101" Nov 01 00:50:07.525652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.520512 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf" Nov 01 00:50:07.575650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.575070 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-openshift-ca\") pod \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " Nov 01 00:50:07.575650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.575253 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-xks-ca\") pod \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " Nov 01 00:50:07.575650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.575354 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72699e9-14fb-4cd4-90ac-676afb83f9c9-kube-api-access-gbch2\" (UniqueName: \"kubernetes.io/projected/e72699e9-14fb-4cd4-90ac-676afb83f9c9-kube-api-access-gbch2\") pod \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\" (UID: \"e72699e9-14fb-4cd4-90ac-676afb83f9c9\") " Nov 01 00:50:07.618657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.617126 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-openshift-ca" pod "e72699e9-14fb-4cd4-90ac-676afb83f9c9" (UID: "e72699e9-14fb-4cd4-90ac-676afb83f9c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:07.630643 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.629810 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72699e9-14fb-4cd4-90ac-676afb83f9c9-kube-api-access-gbch2" pod "e72699e9-14fb-4cd4-90ac-676afb83f9c9" (UID: "e72699e9-14fb-4cd4-90ac-676afb83f9c9"). InnerVolumeSpecName "kube-api-access-gbch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:07.677318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.677238 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbch2\" (UniqueName: \"kubernetes.io/projected/e72699e9-14fb-4cd4-90ac-676afb83f9c9-kube-api-access-gbch2\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:07.677318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.677277 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:07.710859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.705153 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-xks-ca" pod "e72699e9-14fb-4cd4-90ac-676afb83f9c9" (UID: "e72699e9-14fb-4cd4-90ac-676afb83f9c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:07.741722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.740719 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 01 00:50:07.803898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.799244 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72699e9-14fb-4cd4-90ac-676afb83f9c9-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:07.974708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.960322 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.974708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.960380 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:07.974708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.960985 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.974708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.961021 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:07.999824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.997701 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:07.999824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:07.997797 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:08.127098 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:08.127059 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz"] Nov 01 00:50:08.166190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:08.166134 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:08.166505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:08.166481 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:08.166707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:08.166670 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:08.634651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:08.627057 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" event={"ID":"ad371433-6a03-4a7b-abe1-21705b9a2ffb","Type":"ContainerStarted","Data":"7f932c4593159c11e79f16b38301e54f43b30eae54deaf3c4a5bb4fc3418bb36"} Nov 01 00:50:08.947119 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:08.947024 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:08.947351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:08.947322 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:09.009014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.008964 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.009339 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.009290 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:09.009899 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.009854 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.010053 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.010033 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:09.699652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.689827 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" event={"ID":"ad371433-6a03-4a7b-abe1-21705b9a2ffb","Type":"ContainerStarted","Data":"2bdba2b4f4c476c2217b65b4510a6c2094bd0c9e6d95be29921dea8dc002352d"} Nov 01 00:50:09.861647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.861248 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podStartSLOduration=4.861229884 podStartE2EDuration="4.861229884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:09.815184394 +0000 UTC m=+1599.794766754" watchObservedRunningTime="2026-11-01 00:50:09.861229884 +0000 UTC m=+1599.840812219" Nov 01 00:50:09.932683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.921286 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:09.932683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.921414 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.932683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.921456 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:09.961710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.948049 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:09.961710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.948107 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:10.003646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.998594 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.003646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.998669 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:10.003646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.999049 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.003646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:09.999104 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:10.255852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.255750 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:10.438369 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.438327 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:10.765255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.765213 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Nov 01 00:50:10.825682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.784763 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:10.825682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.817419 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:50:10.922911 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.922875 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.923357 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.923199 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:10.944657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.944383 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.944657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.944435 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:10.954999 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.954968 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:10.955289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:10.955238 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:11.000667 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.000612 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.000921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.000891 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:11.929595 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.929537 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.930336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.930284 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:11.947139 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.947108 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.947573 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.947540 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 01 00:50:11.958970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.958941 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.959156 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.959126 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 00:50:11.998885 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.998853 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 01 00:50:11.999182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:11.999134 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 01 00:50:12.924814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:12.924774 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:12.925026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:12.924842 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:13.025814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.024969 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:13.026533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.026512 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:13.031519 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.027671 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:50:13.045415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.045338 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:13.045897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.045853 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:13.047589 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.047379 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:50:13.069766 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.069707 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b"] Nov 01 00:50:13.098964 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.098933 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33ce4550-5zg4b"] Nov 01 00:50:13.108266 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.108235 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:13.108734 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.108672 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:13.110062 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.110015 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 00:50:13.127165 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:50:13.127088 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:50:13.919691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.919654 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:13.919862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:13.919723 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:14.197696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.197217 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88fcc7c7-109d-425c-a5c9-e33cba1c783e" path="/var/lib/kubelet/pods/88fcc7c7-109d-425c-a5c9-e33cba1c783e/volumes" Nov 01 00:50:14.606410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.606382 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn"] Nov 01 00:50:14.607021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.607008 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c9e800fb-e72f-4961-8f97-88baad1a6f4d" containerName="revoke-keys" Nov 01 00:50:14.607082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.607023 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e800fb-e72f-4961-8f97-88baad1a6f4d" containerName="revoke-keys" Nov 01 00:50:14.607082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.607045 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e72699e9-14fb-4cd4-90ac-676afb83f9c9" containerName="revoke-keys" Nov 01 00:50:14.607082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.607050 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72699e9-14fb-4cd4-90ac-676afb83f9c9" containerName="revoke-keys" Nov 01 00:50:14.607171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.607160 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e800fb-e72f-4961-8f97-88baad1a6f4d" containerName="revoke-keys" Nov 01 00:50:14.607171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.607168 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72699e9-14fb-4cd4-90ac-676afb83f9c9" containerName="revoke-keys" Nov 01 00:50:14.611972 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.611952 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.673650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.672496 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:50:14.673650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.673252 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:50:14.689160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.688949 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn"] Nov 01 00:50:14.690708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.690680 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42hq\" (UniqueName: \"kubernetes.io/projected/c99943c0-c763-4e6c-b773-fd27ab1601bc-kube-api-access-s42hq\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.690944 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.690921 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.691089 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.690994 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.792291 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.792257 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42hq\" (UniqueName: \"kubernetes.io/projected/c99943c0-c763-4e6c-b773-fd27ab1601bc-kube-api-access-s42hq\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.792473 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.792412 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.792473 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.792447 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.793126 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.793100 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.794993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.794963 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.879698 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.878453 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42hq\" (UniqueName: \"kubernetes.io/projected/c99943c0-c763-4e6c-b773-fd27ab1601bc-kube-api-access-s42hq\") pod \"maas-api-revoke-sub-b3b26548-bprpn\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:14.919581 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.919549 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:14.919777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.919655 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:14.926571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:14.926548 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:15.766444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:15.766408 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn"] Nov 01 00:50:15.942346 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:15.942313 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:15.942596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:15.942569 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:16.093974 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:16.093889 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" event={"ID":"c99943c0-c763-4e6c-b773-fd27ab1601bc","Type":"ContainerStarted","Data":"622e8db6990d31e2cf013926f474a194dd526826d0b6f0be06ee0ffb9baffe7e"} Nov 01 00:50:16.919038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:16.918609 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:16.919038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:16.918685 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:17.013082 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.012733 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.117364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.117330 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" event={"ID":"c99943c0-c763-4e6c-b773-fd27ab1601bc","Type":"ContainerStarted","Data":"4aca68c283009c9ef1306512f8c186f1507c87641a646ca2760ea7865ec30d11"} Nov 01 00:50:17.190779 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.189694 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:50:17.261763 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.261698 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" podStartSLOduration=3.261679061 podStartE2EDuration="3.261679061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:17.261035705 +0000 UTC m=+1607.240618043" watchObservedRunningTime="2026-11-01 00:50:17.261679061 +0000 UTC m=+1607.241261397" Nov 01 00:50:17.271792 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.271702 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.401039 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.400719 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.416543 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.415720 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.471651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.458326 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.657180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.657134 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.775840 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.775793 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.924711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.856789 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:17.925342 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.921092 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:17.925486 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:17.925460 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:18.257454 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.229855 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:50:18.303683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.274751 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:18.323651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.310847 4910 generic.go:356] "Generic (PLEG): container finished" podID="c99943c0-c763-4e6c-b773-fd27ab1601bc" containerID="4aca68c283009c9ef1306512f8c186f1507c87641a646ca2760ea7865ec30d11" exitCode=0 Nov 01 00:50:18.382649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.382305 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:50:18.382649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.382351 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" event={"ID":"c99943c0-c763-4e6c-b773-fd27ab1601bc","Type":"ContainerDied","Data":"4aca68c283009c9ef1306512f8c186f1507c87641a646ca2760ea7865ec30d11"} Nov 01 00:50:18.382649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.382422 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:50:18.416746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.416701 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:50:18.442650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.417429 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:50:18.713192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.713087 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:50:18.719134 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.719105 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:50:18.770115 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.762714 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 01 00:50:18.884817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.884757 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 01 00:50:18.885209 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.885185 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:50:18.935802 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.935768 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 01 00:50:18.939260 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:18.939195 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 01 00:50:19.074671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:19.063882 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:19.074671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:19.073179 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:50:19.231467 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:19.231016 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:19.258853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:19.258826 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:50:19.418538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:19.417152 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:50:19.682812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:19.682652 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:50:20.036650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.033770 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.067921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.065863 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.067921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.066354 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:50:20.433698 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.411724 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.456603 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.430117 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:50:20.460720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.431036 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.461458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.431230 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:50:20.687334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.687231 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:20.716811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.716785 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:20.944713 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.939001 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-openshift-ca\") pod \"c99943c0-c763-4e6c-b773-fd27ab1601bc\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " Nov 01 00:50:20.945917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.945886 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99943c0-c763-4e6c-b773-fd27ab1601bc-kube-api-access-s42hq\" (UniqueName: \"kubernetes.io/projected/c99943c0-c763-4e6c-b773-fd27ab1601bc-kube-api-access-s42hq\") pod \"c99943c0-c763-4e6c-b773-fd27ab1601bc\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " Nov 01 00:50:20.946691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:20.939610 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-openshift-ca" pod "c99943c0-c763-4e6c-b773-fd27ab1601bc" (UID: "c99943c0-c763-4e6c-b773-fd27ab1601bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:21.039442 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.039389 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:21.195608 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.194798 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:50:21.206467 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.206438 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-xks-ca\") pod \"c99943c0-c763-4e6c-b773-fd27ab1601bc\" (UID: \"c99943c0-c763-4e6c-b773-fd27ab1601bc\") " Nov 01 00:50:21.221242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.208972 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:50:21.221242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.217666 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-xks-ca" pod "c99943c0-c763-4e6c-b773-fd27ab1601bc" (UID: "c99943c0-c763-4e6c-b773-fd27ab1601bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:21.226000 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.224500 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:21.226000 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.224532 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99943c0-c763-4e6c-b773-fd27ab1601bc-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:21.228984 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.228812 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99943c0-c763-4e6c-b773-fd27ab1601bc-kube-api-access-s42hq" pod "c99943c0-c763-4e6c-b773-fd27ab1601bc" (UID: "c99943c0-c763-4e6c-b773-fd27ab1601bc"). InnerVolumeSpecName "kube-api-access-s42hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:21.238774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.236180 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l"] Nov 01 00:50:21.238774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.237197 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c99943c0-c763-4e6c-b773-fd27ab1601bc" containerName="revoke-keys" Nov 01 00:50:21.238774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.237215 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99943c0-c763-4e6c-b773-fd27ab1601bc" containerName="revoke-keys" Nov 01 00:50:21.238774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.237395 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99943c0-c763-4e6c-b773-fd27ab1601bc" containerName="revoke-keys" Nov 01 00:50:21.241518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.240848 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:21.241518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.240896 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:21.241794 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.241766 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.285412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.285308 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l"] Nov 01 00:50:21.315415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.313685 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7"] Nov 01 00:50:21.322146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.322115 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7"] Nov 01 00:50:21.322323 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.322224 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.326720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.325671 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.326720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.325862 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9x8t\" (UniqueName: \"kubernetes.io/projected/3a0d5a22-2261-4e21-aa95-60ece65bacf0-kube-api-access-m9x8t\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.326720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.325885 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.326720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.325953 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42hq\" (UniqueName: \"kubernetes.io/projected/c99943c0-c763-4e6c-b773-fd27ab1601bc-kube-api-access-s42hq\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:21.335600 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.335548 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9"] Nov 01 00:50:21.343323 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.343292 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d8d10c-52jk9"] Nov 01 00:50:21.426928 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.426892 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9x8t\" (UniqueName: \"kubernetes.io/projected/3a0d5a22-2261-4e21-aa95-60ece65bacf0-kube-api-access-m9x8t\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.427135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.426949 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.427135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.426981 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc96513-1914-411e-a293-4e8683794d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.427135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.427043 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkns8\" (UniqueName: \"kubernetes.io/projected/9cc96513-1914-411e-a293-4e8683794d34-kube-api-access-lkns8\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.427911 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.427143 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.427911 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.427245 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc96513-1914-411e-a293-4e8683794d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.428681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.428594 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.436327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.436300 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.488785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.488706 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:21.489455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.489428 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:21.490519 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.490498 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 00:50:21.491664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.491614 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9x8t\" (UniqueName: \"kubernetes.io/projected/3a0d5a22-2261-4e21-aa95-60ece65bacf0-kube-api-access-m9x8t\") pod \"maas-api-revoke-sub-ce1fb8a0-8lr6l\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.528725 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.528693 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc96513-1914-411e-a293-4e8683794d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.528920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.528900 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc96513-1914-411e-a293-4e8683794d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.528988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.528956 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkns8\" (UniqueName: \"kubernetes.io/projected/9cc96513-1914-411e-a293-4e8683794d34-kube-api-access-lkns8\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.529445 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.529422 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc96513-1914-411e-a293-4e8683794d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.531609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.531587 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc96513-1914-411e-a293-4e8683794d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.598245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.598213 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:21.599838 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.599098 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkns8\" (UniqueName: \"kubernetes.io/projected/9cc96513-1914-411e-a293-4e8683794d34-kube-api-access-lkns8\") pod \"maas-api-revoke-sub-37f1902c-m9lc7\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.639745 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.639501 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:21.731065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.730730 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" event={"ID":"c99943c0-c763-4e6c-b773-fd27ab1601bc","Type":"ContainerDied","Data":"622e8db6990d31e2cf013926f474a194dd526826d0b6f0be06ee0ffb9baffe7e"} Nov 01 00:50:21.731065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.730815 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="622e8db6990d31e2cf013926f474a194dd526826d0b6f0be06ee0ffb9baffe7e" Nov 01 00:50:21.731065 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:21.730751 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn" Nov 01 00:50:22.022140 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:22.022110 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0d5a22_2261_4e21_aa95_60ece65bacf0.slice/crio-7b135dd4d4e79a0d8fe7777cc8f442f999ac4167af018c4a84175119ea6d22da WatchSource:0}: Error finding container 7b135dd4d4e79a0d8fe7777cc8f442f999ac4167af018c4a84175119ea6d22da: Status 404 returned error can't find the container with id 7b135dd4d4e79a0d8fe7777cc8f442f999ac4167af018c4a84175119ea6d22da Nov 01 00:50:22.023233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.023207 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l"] Nov 01 00:50:22.077077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.077029 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7"] Nov 01 00:50:22.195662 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.195608 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2b84cb-54c9-4b22-9118-a0e379c70fde" path="/var/lib/kubelet/pods/bd2b84cb-54c9-4b22-9118-a0e379c70fde/volumes" Nov 01 00:50:22.541129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.541088 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg"] Nov 01 00:50:22.549054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.548986 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.606086 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.606056 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg"] Nov 01 00:50:22.648668 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.648617 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64q7g\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-kube-api-access-64q7g\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.648823 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.648727 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.648823 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.648771 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.648823 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.648812 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.649027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.648936 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.649120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.649100 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.649168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.649131 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.649168 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.649152 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.649248 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.649231 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.740559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.740524 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" event={"ID":"9cc96513-1914-411e-a293-4e8683794d34","Type":"ContainerStarted","Data":"178412defffbe2bdab24443fe8631ff5e9debbce45fbf2cfb3608e974e2f38a5"} Nov 01 00:50:22.740798 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.740595 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" event={"ID":"9cc96513-1914-411e-a293-4e8683794d34","Type":"ContainerStarted","Data":"4a2ca0dec8fc9aa1a4e4dcc82543778d64758d1f66ee47351d311aa8bbd188e5"} Nov 01 00:50:22.742308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.742275 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" event={"ID":"3a0d5a22-2261-4e21-aa95-60ece65bacf0","Type":"ContainerStarted","Data":"0f5d00dc77d3970f12c9a1d1c08fe525807d2952b99998190210bf05cbcab3a1"} Nov 01 00:50:22.742434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.742331 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" event={"ID":"3a0d5a22-2261-4e21-aa95-60ece65bacf0","Type":"ContainerStarted","Data":"7b135dd4d4e79a0d8fe7777cc8f442f999ac4167af018c4a84175119ea6d22da"} Nov 01 00:50:22.750648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750602 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.750774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750668 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.750774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.750774 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750742 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.750952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750859 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64q7g\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-kube-api-access-64q7g\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.750952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750895 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.750952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750942 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.751104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.750996 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.751104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.751031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.751585 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.751560 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-data\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.752189 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.752167 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-certs\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.752291 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.752212 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-credential-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.752351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.752320 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istiod-ca-cert\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.752792 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.752765 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-socket\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.754642 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.754604 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-envoy\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.755758 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.755739 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-podinfo\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.812474 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.812408 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-token\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.812474 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.812453 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64q7g\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-kube-api-access-64q7g\") pod \"e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.860377 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.860320 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" podStartSLOduration=2.86030178 podStartE2EDuration="2.86030178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:22.856675147 +0000 UTC m=+1612.836257482" watchObservedRunningTime="2026-11-01 00:50:22.86030178 +0000 UTC m=+1612.839884118" Nov 01 00:50:22.869308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.869278 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:22.979887 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:22.979824 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" podStartSLOduration=2.979805764 podStartE2EDuration="2.979805764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:22.971482116 +0000 UTC m=+1612.951064454" watchObservedRunningTime="2026-11-01 00:50:22.979805764 +0000 UTC m=+1612.959388099" Nov 01 00:50:23.100731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.100642 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk"] Nov 01 00:50:23.106146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.106128 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.157954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.157293 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.157954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.157419 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.157954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.157648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjh2m\" (UniqueName: \"kubernetes.io/projected/48258d78-1ace-43f7-bbab-661e80719f33-kube-api-access-sjh2m\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.165329 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.165300 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk"] Nov 01 00:50:23.259031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.258991 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjh2m\" (UniqueName: \"kubernetes.io/projected/48258d78-1ace-43f7-bbab-661e80719f33-kube-api-access-sjh2m\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.259215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.259097 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.259215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.259203 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.260078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.260048 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.262063 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.262037 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.290792 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:23.290760 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5abcdca_f9e8_4f6a_b644_34df1bfe236b.slice/crio-5425e10ce07e606da95bcf9b469dac3535f8e23b5b86afe88d3c19e988c1ede8 WatchSource:0}: Error finding container 5425e10ce07e606da95bcf9b469dac3535f8e23b5b86afe88d3c19e988c1ede8: Status 404 returned error can't find the container with id 5425e10ce07e606da95bcf9b469dac3535f8e23b5b86afe88d3c19e988c1ede8 Nov 01 00:50:23.291480 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.291449 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg"] Nov 01 00:50:23.294213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.294151 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:23.294301 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.294269 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:23.294338 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.294315 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:23.320354 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.320334 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjh2m\" (UniqueName: \"kubernetes.io/projected/48258d78-1ace-43f7-bbab-661e80719f33-kube-api-access-sjh2m\") pod \"maas-api-revoke-sub-0b16fa68-f98rk\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.518998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.518971 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:23.755514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.755480 4910 generic.go:356] "Generic (PLEG): container finished" podID="9cc96513-1914-411e-a293-4e8683794d34" containerID="178412defffbe2bdab24443fe8631ff5e9debbce45fbf2cfb3608e974e2f38a5" exitCode=0 Nov 01 00:50:23.755729 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.755573 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" event={"ID":"9cc96513-1914-411e-a293-4e8683794d34","Type":"ContainerDied","Data":"178412defffbe2bdab24443fe8631ff5e9debbce45fbf2cfb3608e974e2f38a5"} Nov 01 00:50:23.757646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.757599 4910 generic.go:356] "Generic (PLEG): container finished" podID="3a0d5a22-2261-4e21-aa95-60ece65bacf0" containerID="0f5d00dc77d3970f12c9a1d1c08fe525807d2952b99998190210bf05cbcab3a1" exitCode=0 Nov 01 00:50:23.757769 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.757697 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" event={"ID":"3a0d5a22-2261-4e21-aa95-60ece65bacf0","Type":"ContainerDied","Data":"0f5d00dc77d3970f12c9a1d1c08fe525807d2952b99998190210bf05cbcab3a1"} Nov 01 00:50:23.759913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.759891 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" event={"ID":"b5abcdca-f9e8-4f6a-b644-34df1bfe236b","Type":"ContainerStarted","Data":"6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b"} Nov 01 00:50:23.760051 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.759933 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" event={"ID":"b5abcdca-f9e8-4f6a-b644-34df1bfe236b","Type":"ContainerStarted","Data":"5425e10ce07e606da95bcf9b469dac3535f8e23b5b86afe88d3c19e988c1ede8"} Nov 01 00:50:23.870398 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.870306 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:23.903503 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:23.903477 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48258d78_1ace_43f7_bbab_661e80719f33.slice/crio-2786f0c9f5fdcef3bd88654aab4bb25f4c02eae5e4cf9eb5e4fc315d1d6162a8 WatchSource:0}: Error finding container 2786f0c9f5fdcef3bd88654aab4bb25f4c02eae5e4cf9eb5e4fc315d1d6162a8: Status 404 returned error can't find the container with id 2786f0c9f5fdcef3bd88654aab4bb25f4c02eae5e4cf9eb5e4fc315d1d6162a8 Nov 01 00:50:23.905174 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.905154 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk"] Nov 01 00:50:23.996078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:23.996010 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podStartSLOduration=1.99599019 podStartE2EDuration="1.99599019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:23.992481177 +0000 UTC m=+1613.972063530" watchObservedRunningTime="2026-11-01 00:50:23.99599019 +0000 UTC m=+1613.975572527" Nov 01 00:50:24.800240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:24.800198 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" event={"ID":"48258d78-1ace-43f7-bbab-661e80719f33","Type":"ContainerStarted","Data":"72efdfda95df7585ea107baec4ad47ad1a8f28116c72494cdcee91363e7dc3c3"} Nov 01 00:50:24.800766 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:24.800299 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" event={"ID":"48258d78-1ace-43f7-bbab-661e80719f33","Type":"ContainerStarted","Data":"2786f0c9f5fdcef3bd88654aab4bb25f4c02eae5e4cf9eb5e4fc315d1d6162a8"} Nov 01 00:50:24.877750 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:24.877363 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 00:50:24.877750 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:24.877431 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 00:50:24.926906 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:24.926858 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" podStartSLOduration=1.926845466 podStartE2EDuration="1.926845466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:24.925360743 +0000 UTC m=+1614.904943080" watchObservedRunningTime="2026-11-01 00:50:24.926845466 +0000 UTC m=+1614.906428232" Nov 01 00:50:25.113214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.113190 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:25.116384 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.116365 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:25.201702 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.201674 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc96513-1914-411e-a293-4e8683794d34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc96513-1914-411e-a293-4e8683794d34-maas-api-xks-ca\") pod \"9cc96513-1914-411e-a293-4e8683794d34\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " Nov 01 00:50:25.201925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.201842 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc96513-1914-411e-a293-4e8683794d34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc96513-1914-411e-a293-4e8683794d34-maas-api-openshift-ca\") pod \"9cc96513-1914-411e-a293-4e8683794d34\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " Nov 01 00:50:25.202003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.201963 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0d5a22-2261-4e21-aa95-60ece65bacf0-kube-api-access-m9x8t\" (UniqueName: \"kubernetes.io/projected/3a0d5a22-2261-4e21-aa95-60ece65bacf0-kube-api-access-m9x8t\") pod \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " Nov 01 00:50:25.202062 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.202006 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-xks-ca\") pod \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " Nov 01 00:50:25.202118 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.202057 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc96513-1914-411e-a293-4e8683794d34-kube-api-access-lkns8\" (UniqueName: \"kubernetes.io/projected/9cc96513-1914-411e-a293-4e8683794d34-kube-api-access-lkns8\") pod \"9cc96513-1914-411e-a293-4e8683794d34\" (UID: \"9cc96513-1914-411e-a293-4e8683794d34\") " Nov 01 00:50:25.202118 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.202107 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-openshift-ca\") pod \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\" (UID: \"3a0d5a22-2261-4e21-aa95-60ece65bacf0\") " Nov 01 00:50:25.202211 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.202186 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc96513-1914-411e-a293-4e8683794d34-maas-api-openshift-ca" pod "9cc96513-1914-411e-a293-4e8683794d34" (UID: "9cc96513-1914-411e-a293-4e8683794d34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:25.202903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.202878 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc96513-1914-411e-a293-4e8683794d34-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:25.203327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.203298 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-openshift-ca" pod "3a0d5a22-2261-4e21-aa95-60ece65bacf0" (UID: "3a0d5a22-2261-4e21-aa95-60ece65bacf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:25.204512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.204481 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0d5a22-2261-4e21-aa95-60ece65bacf0-kube-api-access-m9x8t" pod "3a0d5a22-2261-4e21-aa95-60ece65bacf0" (UID: "3a0d5a22-2261-4e21-aa95-60ece65bacf0"). InnerVolumeSpecName "kube-api-access-m9x8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:25.204607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.204528 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc96513-1914-411e-a293-4e8683794d34-maas-api-xks-ca" pod "9cc96513-1914-411e-a293-4e8683794d34" (UID: "9cc96513-1914-411e-a293-4e8683794d34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:25.204682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.204665 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-xks-ca" pod "3a0d5a22-2261-4e21-aa95-60ece65bacf0" (UID: "3a0d5a22-2261-4e21-aa95-60ece65bacf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:25.205076 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.205058 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc96513-1914-411e-a293-4e8683794d34-kube-api-access-lkns8" pod "9cc96513-1914-411e-a293-4e8683794d34" (UID: "9cc96513-1914-411e-a293-4e8683794d34"). InnerVolumeSpecName "kube-api-access-lkns8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:25.303942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.303858 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9x8t\" (UniqueName: \"kubernetes.io/projected/3a0d5a22-2261-4e21-aa95-60ece65bacf0-kube-api-access-m9x8t\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:25.303942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.303886 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:25.303942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.303897 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkns8\" (UniqueName: \"kubernetes.io/projected/9cc96513-1914-411e-a293-4e8683794d34-kube-api-access-lkns8\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:25.303942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.303906 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0d5a22-2261-4e21-aa95-60ece65bacf0-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:25.303942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.303915 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc96513-1914-411e-a293-4e8683794d34-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:25.811343 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.811315 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" Nov 01 00:50:25.811781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.811354 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7" event={"ID":"9cc96513-1914-411e-a293-4e8683794d34","Type":"ContainerDied","Data":"4a2ca0dec8fc9aa1a4e4dcc82543778d64758d1f66ee47351d311aa8bbd188e5"} Nov 01 00:50:25.811781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.811403 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2ca0dec8fc9aa1a4e4dcc82543778d64758d1f66ee47351d311aa8bbd188e5" Nov 01 00:50:25.812802 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.812785 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" Nov 01 00:50:25.812901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.812829 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l" event={"ID":"3a0d5a22-2261-4e21-aa95-60ece65bacf0","Type":"ContainerDied","Data":"7b135dd4d4e79a0d8fe7777cc8f442f999ac4167af018c4a84175119ea6d22da"} Nov 01 00:50:25.812901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.812874 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b135dd4d4e79a0d8fe7777cc8f442f999ac4167af018c4a84175119ea6d22da" Nov 01 00:50:25.814600 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.814579 4910 generic.go:356] "Generic (PLEG): container finished" podID="48258d78-1ace-43f7-bbab-661e80719f33" containerID="72efdfda95df7585ea107baec4ad47ad1a8f28116c72494cdcee91363e7dc3c3" exitCode=0 Nov 01 00:50:25.814722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.814669 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" event={"ID":"48258d78-1ace-43f7-bbab-661e80719f33","Type":"ContainerDied","Data":"72efdfda95df7585ea107baec4ad47ad1a8f28116c72494cdcee91363e7dc3c3"} Nov 01 00:50:25.875651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.874080 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 00:50:25.875651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:25.874144 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 00:50:26.794478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.794329 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj"] Nov 01 00:50:26.795367 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.795342 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9cc96513-1914-411e-a293-4e8683794d34" containerName="revoke-keys" Nov 01 00:50:26.795478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.795368 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc96513-1914-411e-a293-4e8683794d34" containerName="revoke-keys" Nov 01 00:50:26.795478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.795418 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3a0d5a22-2261-4e21-aa95-60ece65bacf0" containerName="revoke-keys" Nov 01 00:50:26.795478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.795425 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0d5a22-2261-4e21-aa95-60ece65bacf0" containerName="revoke-keys" Nov 01 00:50:26.795685 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.795649 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0d5a22-2261-4e21-aa95-60ece65bacf0" containerName="revoke-keys" Nov 01 00:50:26.795685 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.795677 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc96513-1914-411e-a293-4e8683794d34" containerName="revoke-keys" Nov 01 00:50:26.799858 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.799834 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.834348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.834318 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.834684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.834376 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.834684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.834506 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xd2f\" (UniqueName: \"kubernetes.io/projected/8a0e9f00-2649-486f-9416-3c285d530a6c-kube-api-access-9xd2f\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.890651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.887082 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj"] Nov 01 00:50:26.904152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.903700 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 00:50:26.904152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.903786 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 00:50:26.936459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.936415 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.936657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.936471 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.936657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.936522 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xd2f\" (UniqueName: \"kubernetes.io/projected/8a0e9f00-2649-486f-9416-3c285d530a6c-kube-api-access-9xd2f\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.938666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.937279 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:26.939751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:26.939727 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:27.006411 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.006374 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xd2f\" (UniqueName: \"kubernetes.io/projected/8a0e9f00-2649-486f-9416-3c285d530a6c-kube-api-access-9xd2f\") pod \"maas-api-revoke-sub-a459d458-w6rcj\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:27.101263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.101177 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k"] Nov 01 00:50:27.111510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.111482 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3156970-msj4k"] Nov 01 00:50:27.114394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.114376 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:27.196933 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.196709 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:27.240068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.240045 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48258d78-1ace-43f7-bbab-661e80719f33-kube-api-access-sjh2m\" (UniqueName: \"kubernetes.io/projected/48258d78-1ace-43f7-bbab-661e80719f33-kube-api-access-sjh2m\") pod \"48258d78-1ace-43f7-bbab-661e80719f33\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " Nov 01 00:50:27.240293 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.240278 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-xks-ca\") pod \"48258d78-1ace-43f7-bbab-661e80719f33\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " Nov 01 00:50:27.240504 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.240487 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-openshift-ca\") pod \"48258d78-1ace-43f7-bbab-661e80719f33\" (UID: \"48258d78-1ace-43f7-bbab-661e80719f33\") " Nov 01 00:50:27.240877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.240852 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-openshift-ca" pod "48258d78-1ace-43f7-bbab-661e80719f33" (UID: "48258d78-1ace-43f7-bbab-661e80719f33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:27.244666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.243485 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:27.244666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.243730 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-xks-ca" pod "48258d78-1ace-43f7-bbab-661e80719f33" (UID: "48258d78-1ace-43f7-bbab-661e80719f33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:27.244666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.244466 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48258d78-1ace-43f7-bbab-661e80719f33-kube-api-access-sjh2m" pod "48258d78-1ace-43f7-bbab-661e80719f33" (UID: "48258d78-1ace-43f7-bbab-661e80719f33"). InnerVolumeSpecName "kube-api-access-sjh2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:27.345364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.345328 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjh2m\" (UniqueName: \"kubernetes.io/projected/48258d78-1ace-43f7-bbab-661e80719f33-kube-api-access-sjh2m\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:27.345364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.345367 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48258d78-1ace-43f7-bbab-661e80719f33-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:27.582650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.577525 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj"] Nov 01 00:50:27.610253 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:27.610214 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0e9f00_2649_486f_9416_3c285d530a6c.slice/crio-49402c2c87f062241f906f86a9210a6c3fcb066885e26e3ab065abc6c3be04ba WatchSource:0}: Error finding container 49402c2c87f062241f906f86a9210a6c3fcb066885e26e3ab065abc6c3be04ba: Status 404 returned error can't find the container with id 49402c2c87f062241f906f86a9210a6c3fcb066885e26e3ab065abc6c3be04ba Nov 01 00:50:27.835954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.835863 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" event={"ID":"8a0e9f00-2649-486f-9416-3c285d530a6c","Type":"ContainerStarted","Data":"49402c2c87f062241f906f86a9210a6c3fcb066885e26e3ab065abc6c3be04ba"} Nov 01 00:50:27.837639 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.837602 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" Nov 01 00:50:27.837754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.837669 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk" event={"ID":"48258d78-1ace-43f7-bbab-661e80719f33","Type":"ContainerDied","Data":"2786f0c9f5fdcef3bd88654aab4bb25f4c02eae5e4cf9eb5e4fc315d1d6162a8"} Nov 01 00:50:27.837754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.837710 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2786f0c9f5fdcef3bd88654aab4bb25f4c02eae5e4cf9eb5e4fc315d1d6162a8" Nov 01 00:50:27.881265 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.881234 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 00:50:27.881467 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:27.881440 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 00:50:28.220207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.220177 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5b18f4-2e9c-46f0-b439-d95b6e4161ef" path="/var/lib/kubelet/pods/5d5b18f4-2e9c-46f0-b439-d95b6e4161ef/volumes" Nov 01 00:50:28.686294 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.686250 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc"] Nov 01 00:50:28.687339 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.687316 4910 cpu_manager.go:414] "containerMap: removing container" podUID="48258d78-1ace-43f7-bbab-661e80719f33" containerName="revoke-keys" Nov 01 00:50:28.687422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.687344 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48258d78-1ace-43f7-bbab-661e80719f33" containerName="revoke-keys" Nov 01 00:50:28.687591 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.687576 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="48258d78-1ace-43f7-bbab-661e80719f33" containerName="revoke-keys" Nov 01 00:50:28.692323 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.692260 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.745913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.745888 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc"] Nov 01 00:50:28.807708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.807667 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.807866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.807797 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwln\" (UniqueName: \"kubernetes.io/projected/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-kube-api-access-tqwln\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.807998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.807979 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.848685 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.848647 4910 generic.go:356] "Generic (PLEG): container finished" podID="8a0e9f00-2649-486f-9416-3c285d530a6c" containerID="5e761aa86022e8ed0d0b6514209bbb249b205faf12a2edc3f87f4b2ad6c041e3" exitCode=0 Nov 01 00:50:28.849091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.848774 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" event={"ID":"8a0e9f00-2649-486f-9416-3c285d530a6c","Type":"ContainerDied","Data":"5e761aa86022e8ed0d0b6514209bbb249b205faf12a2edc3f87f4b2ad6c041e3"} Nov 01 00:50:28.880644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.879782 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 00:50:28.880644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.879941 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 00:50:28.909134 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.909094 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.909313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.909186 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwln\" (UniqueName: \"kubernetes.io/projected/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-kube-api-access-tqwln\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.909378 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.909341 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.910040 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.910017 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:28.911849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:28.911819 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:29.013639 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.013594 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwln\" (UniqueName: \"kubernetes.io/projected/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-kube-api-access-tqwln\") pod \"maas-api-revoke-sub-13b95067-4prjc\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:29.048991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.048963 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf"] Nov 01 00:50:29.083789 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.083754 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820d88bb-68khf"] Nov 01 00:50:29.313924 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.313837 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:29.716466 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:29.716435 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ae5490_5a25_4d8a_b9cb_38e4cdb2f07d.slice/crio-6fbe3d8e7def90aa28e45fdb84c193ea764de6e83170ff3710278189be01983c WatchSource:0}: Error finding container 6fbe3d8e7def90aa28e45fdb84c193ea764de6e83170ff3710278189be01983c: Status 404 returned error can't find the container with id 6fbe3d8e7def90aa28e45fdb84c193ea764de6e83170ff3710278189be01983c Nov 01 00:50:29.716987 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.716966 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc"] Nov 01 00:50:29.863484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.863437 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" event={"ID":"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d","Type":"ContainerStarted","Data":"6fbe3d8e7def90aa28e45fdb84c193ea764de6e83170ff3710278189be01983c"} Nov 01 00:50:29.870345 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.870318 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 00:50:29.870492 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:29.870377 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 00:50:30.137787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.137759 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:30.194844 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.194808 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027eb9e0-0d3f-41cb-b367-6581832f1f77" path="/var/lib/kubelet/pods/027eb9e0-0d3f-41cb-b367-6581832f1f77/volumes" Nov 01 00:50:30.228869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.228843 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-openshift-ca\") pod \"8a0e9f00-2649-486f-9416-3c285d530a6c\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " Nov 01 00:50:30.229056 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.228906 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-xks-ca\") pod \"8a0e9f00-2649-486f-9416-3c285d530a6c\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " Nov 01 00:50:30.229056 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.229023 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0e9f00-2649-486f-9416-3c285d530a6c-kube-api-access-9xd2f\" (UniqueName: \"kubernetes.io/projected/8a0e9f00-2649-486f-9416-3c285d530a6c-kube-api-access-9xd2f\") pod \"8a0e9f00-2649-486f-9416-3c285d530a6c\" (UID: \"8a0e9f00-2649-486f-9416-3c285d530a6c\") " Nov 01 00:50:30.229252 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.229229 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-openshift-ca" pod "8a0e9f00-2649-486f-9416-3c285d530a6c" (UID: "8a0e9f00-2649-486f-9416-3c285d530a6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:30.229440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.229423 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:30.231073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.231044 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-xks-ca" pod "8a0e9f00-2649-486f-9416-3c285d530a6c" (UID: "8a0e9f00-2649-486f-9416-3c285d530a6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:30.231185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.231167 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0e9f00-2649-486f-9416-3c285d530a6c-kube-api-access-9xd2f" pod "8a0e9f00-2649-486f-9416-3c285d530a6c" (UID: "8a0e9f00-2649-486f-9416-3c285d530a6c"). InnerVolumeSpecName "kube-api-access-9xd2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:30.324805 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.324726 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc"] Nov 01 00:50:30.325944 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.325917 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8a0e9f00-2649-486f-9416-3c285d530a6c" containerName="revoke-keys" Nov 01 00:50:30.325944 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.325942 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0e9f00-2649-486f-9416-3c285d530a6c" containerName="revoke-keys" Nov 01 00:50:30.326166 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.326151 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0e9f00-2649-486f-9416-3c285d530a6c" containerName="revoke-keys" Nov 01 00:50:30.331384 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.331355 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e9f00-2649-486f-9416-3c285d530a6c-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:30.331558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.331540 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xd2f\" (UniqueName: \"kubernetes.io/projected/8a0e9f00-2649-486f-9416-3c285d530a6c-kube-api-access-9xd2f\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:30.335043 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.335021 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.387825 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.386165 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc"] Nov 01 00:50:30.433008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.432971 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.433171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.433131 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86bq\" (UniqueName: \"kubernetes.io/projected/98cf91fa-3388-404e-8d04-4f3352cc1eb0-kube-api-access-p86bq\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.433269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.433233 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.444405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.444378 4910 scope.go:122] "RemoveContainer" containerID="e08a8bc5e20cbd406166db254fbf0cdb50039ceaaeb4c05e695400fcee8416a0" Nov 01 00:50:30.458107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.458085 4910 scope.go:122] "RemoveContainer" containerID="09d0cbc31753a1809d83cae21d32437f8038eb57943ed223144cc54fed50e347" Nov 01 00:50:30.468914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.468895 4910 scope.go:122] "RemoveContainer" containerID="3698bde14f2bc731c7817921aa5bace28ac4bf2935d9ac5756d2e4411653b316" Nov 01 00:50:30.534908 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.534876 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.535110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.534988 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86bq\" (UniqueName: \"kubernetes.io/projected/98cf91fa-3388-404e-8d04-4f3352cc1eb0-kube-api-access-p86bq\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.535110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.535061 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.535583 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.535560 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.537812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.537785 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.597137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.597064 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86bq\" (UniqueName: \"kubernetes.io/projected/98cf91fa-3388-404e-8d04-4f3352cc1eb0-kube-api-access-p86bq\") pod \"maas-api-revoke-sub-eb1daa24-qjqxc\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.655982 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.655951 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:30.876416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.876324 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" event={"ID":"8a0e9f00-2649-486f-9416-3c285d530a6c","Type":"ContainerDied","Data":"49402c2c87f062241f906f86a9210a6c3fcb066885e26e3ab065abc6c3be04ba"} Nov 01 00:50:30.876416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.876389 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49402c2c87f062241f906f86a9210a6c3fcb066885e26e3ab065abc6c3be04ba" Nov 01 00:50:30.876416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.876391 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj" Nov 01 00:50:30.878596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.878567 4910 generic.go:356] "Generic (PLEG): container finished" podID="f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" containerID="ecae26c05777c20f424a3fdcf98b76371ebe3b209ef96f4be6a2e4764443d39e" exitCode=0 Nov 01 00:50:30.878853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.878603 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" event={"ID":"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d","Type":"ContainerDied","Data":"ecae26c05777c20f424a3fdcf98b76371ebe3b209ef96f4be6a2e4764443d39e"} Nov 01 00:50:30.946919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.946876 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:30.947579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.947556 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:30.948457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:30.948438 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:50:31.052218 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:31.052192 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc"] Nov 01 00:50:31.889001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:31.888967 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" event={"ID":"98cf91fa-3388-404e-8d04-4f3352cc1eb0","Type":"ContainerStarted","Data":"7536fbbee52df3786e2edea667908f4d8be604f9bb5ad8a5754c04dbd2189aa8"} Nov 01 00:50:31.889449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:31.889028 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" event={"ID":"98cf91fa-3388-404e-8d04-4f3352cc1eb0","Type":"ContainerStarted","Data":"5f2affa4ddf9c5c8f42619d52e8e58076b641f36ff5e6b58c50f63a8fa22c9d5"} Nov 01 00:50:32.015648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.010418 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" podStartSLOduration=2.010402228 podStartE2EDuration="2.010402228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:32.007674526 +0000 UTC m=+1621.987256885" watchObservedRunningTime="2026-11-01 00:50:32.010402228 +0000 UTC m=+1621.989984570" Nov 01 00:50:32.051204 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.051172 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5"] Nov 01 00:50:32.058496 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.058460 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f71f39-rjdl5"] Nov 01 00:50:32.072513 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.072492 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:32.194690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.194571 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac" path="/var/lib/kubelet/pods/4ce8e33e-5c07-4c64-bd84-7bdc1b2f01ac/volumes" Nov 01 00:50:32.264579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.264548 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-openshift-ca\") pod \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " Nov 01 00:50:32.264804 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.264660 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-xks-ca\") pod \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " Nov 01 00:50:32.264863 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.264817 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-kube-api-access-tqwln\" (UniqueName: \"kubernetes.io/projected/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-kube-api-access-tqwln\") pod \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\" (UID: \"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d\") " Nov 01 00:50:32.264990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.264966 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-openshift-ca" pod "f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" (UID: "f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:32.265405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.265332 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:32.267040 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.267017 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-xks-ca" pod "f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" (UID: "f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:32.267040 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.267031 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-kube-api-access-tqwln" pod "f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" (UID: "f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d"). InnerVolumeSpecName "kube-api-access-tqwln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:32.366528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.366494 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:32.366528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.366529 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqwln\" (UniqueName: \"kubernetes.io/projected/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d-kube-api-access-tqwln\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:32.903150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.903115 4910 generic.go:356] "Generic (PLEG): container finished" podID="98cf91fa-3388-404e-8d04-4f3352cc1eb0" containerID="7536fbbee52df3786e2edea667908f4d8be604f9bb5ad8a5754c04dbd2189aa8" exitCode=0 Nov 01 00:50:32.903618 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.903194 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" event={"ID":"98cf91fa-3388-404e-8d04-4f3352cc1eb0","Type":"ContainerDied","Data":"7536fbbee52df3786e2edea667908f4d8be604f9bb5ad8a5754c04dbd2189aa8"} Nov 01 00:50:32.904990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.904965 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" Nov 01 00:50:32.905119 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.904990 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc" event={"ID":"f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d","Type":"ContainerDied","Data":"6fbe3d8e7def90aa28e45fdb84c193ea764de6e83170ff3710278189be01983c"} Nov 01 00:50:32.905119 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:32.905038 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fbe3d8e7def90aa28e45fdb84c193ea764de6e83170ff3710278189be01983c" Nov 01 00:50:34.055786 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.055747 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7"] Nov 01 00:50:34.101855 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.101822 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b1f7581-ztjs7"] Nov 01 00:50:34.104704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.104681 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:34.189096 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.189010 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-openshift-ca\") pod \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " Nov 01 00:50:34.189258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.189176 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cf91fa-3388-404e-8d04-4f3352cc1eb0-kube-api-access-p86bq\" (UniqueName: \"kubernetes.io/projected/98cf91fa-3388-404e-8d04-4f3352cc1eb0-kube-api-access-p86bq\") pod \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " Nov 01 00:50:34.189349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.189328 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-openshift-ca" pod "98cf91fa-3388-404e-8d04-4f3352cc1eb0" (UID: "98cf91fa-3388-404e-8d04-4f3352cc1eb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:34.189415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.189392 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-xks-ca\") pod \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\" (UID: \"98cf91fa-3388-404e-8d04-4f3352cc1eb0\") " Nov 01 00:50:34.189842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.189813 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:34.191647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.191476 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-xks-ca" pod "98cf91fa-3388-404e-8d04-4f3352cc1eb0" (UID: "98cf91fa-3388-404e-8d04-4f3352cc1eb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:34.191836 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.191817 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cf91fa-3388-404e-8d04-4f3352cc1eb0-kube-api-access-p86bq" pod "98cf91fa-3388-404e-8d04-4f3352cc1eb0" (UID: "98cf91fa-3388-404e-8d04-4f3352cc1eb0"). InnerVolumeSpecName "kube-api-access-p86bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:34.194525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.194499 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e68eb90-0395-4e19-8df6-28408b62b45d" path="/var/lib/kubelet/pods/8e68eb90-0395-4e19-8df6-28408b62b45d/volumes" Nov 01 00:50:34.291569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.291532 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cf91fa-3388-404e-8d04-4f3352cc1eb0-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:34.291569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.291566 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86bq\" (UniqueName: \"kubernetes.io/projected/98cf91fa-3388-404e-8d04-4f3352cc1eb0-kube-api-access-p86bq\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:34.926884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.926853 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" Nov 01 00:50:34.927087 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.926851 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc" event={"ID":"98cf91fa-3388-404e-8d04-4f3352cc1eb0","Type":"ContainerDied","Data":"5f2affa4ddf9c5c8f42619d52e8e58076b641f36ff5e6b58c50f63a8fa22c9d5"} Nov 01 00:50:34.927087 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:34.926934 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2affa4ddf9c5c8f42619d52e8e58076b641f36ff5e6b58c50f63a8fa22c9d5" Nov 01 00:50:36.681067 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.681036 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj"] Nov 01 00:50:36.681752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.681736 4910 cpu_manager.go:414] "containerMap: removing container" podUID="98cf91fa-3388-404e-8d04-4f3352cc1eb0" containerName="revoke-keys" Nov 01 00:50:36.681752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.681753 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cf91fa-3388-404e-8d04-4f3352cc1eb0" containerName="revoke-keys" Nov 01 00:50:36.681895 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.681784 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" containerName="revoke-keys" Nov 01 00:50:36.681895 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.681789 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" containerName="revoke-keys" Nov 01 00:50:36.681986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.681898 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cf91fa-3388-404e-8d04-4f3352cc1eb0" containerName="revoke-keys" Nov 01 00:50:36.681986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.681912 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" containerName="revoke-keys" Nov 01 00:50:36.685735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.685712 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724003 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724173 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724444 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724483 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724606 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724674 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724750 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.724905 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.726380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.725039 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6qw\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-kube-api-access-mx6qw\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826188 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826385 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826277 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826406 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826443 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826496 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826529 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826562 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826668 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826815 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826756 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6qw\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-kube-api-access-mx6qw\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.826943 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826917 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.827005 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.826957 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.827202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.827160 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.827202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.827192 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.827457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.827429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.830040 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.830006 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.830040 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.830022 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.887912 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.887871 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.888071 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.888038 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6qw\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-kube-api-access-mx6qw\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:36.908892 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:36.908858 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj"] Nov 01 00:50:37.001078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.001046 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:37.015112 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.015083 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx"] Nov 01 00:50:37.021543 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.021519 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.065352 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.065319 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc"] Nov 01 00:50:37.076494 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.076457 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1fcf80-r5blc"] Nov 01 00:50:37.086421 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.086385 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx"] Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132403 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rptv\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-kube-api-access-9rptv\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132506 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132552 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132593 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132683 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132807 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132885 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff05228-9e12-407d-b097-afba164aab6c-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.132961 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff05228-9e12-407d-b097-afba164aab6c-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.133334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.133008 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.233982 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234188 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234055 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff05228-9e12-407d-b097-afba164aab6c-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234188 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234112 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff05228-9e12-407d-b097-afba164aab6c-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234188 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234151 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234371 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234314 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rptv\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-kube-api-access-9rptv\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234426 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234410 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234456 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234539 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234494 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234578 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234608 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234892 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234872 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff05228-9e12-407d-b097-afba164aab6c-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.234975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.234915 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.235136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.235115 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.235226 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.235203 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.236971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.236946 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.237443 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.237423 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff05228-9e12-407d-b097-afba164aab6c-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.295882 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.295807 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.296026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.295888 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rptv\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-kube-api-access-9rptv\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.365422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.365386 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:37.487107 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:37.486603 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e0c83e_890c_4a87_888e_93c1f5bbc00e.slice/crio-af3673ef3c91203201f5a43267feec517dc685c226b2d43dac6c847acef2f299 WatchSource:0}: Error finding container af3673ef3c91203201f5a43267feec517dc685c226b2d43dac6c847acef2f299: Status 404 returned error can't find the container with id af3673ef3c91203201f5a43267feec517dc685c226b2d43dac6c847acef2f299 Nov 01 00:50:37.487920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.487892 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj"] Nov 01 00:50:37.490588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.489895 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.490588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.489980 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.490588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.490022 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.781262 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.781230 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx"] Nov 01 00:50:37.787894 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:37.787866 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff05228_9e12_407d_b097_afba164aab6c.slice/crio-ca7cfe2506a8f4c489a5ad16c8017bac4196765639fab1ffdf9d02444b565ee1 WatchSource:0}: Error finding container ca7cfe2506a8f4c489a5ad16c8017bac4196765639fab1ffdf9d02444b565ee1: Status 404 returned error can't find the container with id ca7cfe2506a8f4c489a5ad16c8017bac4196765639fab1ffdf9d02444b565ee1 Nov 01 00:50:37.790744 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.790707 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.790853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.790787 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.790853 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.790834 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:50:37.972674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.968693 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" event={"ID":"20e0c83e-890c-4a87-888e-93c1f5bbc00e","Type":"ContainerStarted","Data":"e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e"} Nov 01 00:50:37.972674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.968790 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" event={"ID":"20e0c83e-890c-4a87-888e-93c1f5bbc00e","Type":"ContainerStarted","Data":"af3673ef3c91203201f5a43267feec517dc685c226b2d43dac6c847acef2f299"} Nov 01 00:50:37.972674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.971373 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" event={"ID":"cff05228-9e12-407d-b097-afba164aab6c","Type":"ContainerStarted","Data":"ddf961362ce532e30700c934fdb01026f6016beae59a489eaf6f93b9e61094b2"} Nov 01 00:50:37.972674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:37.971420 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" event={"ID":"cff05228-9e12-407d-b097-afba164aab6c","Type":"ContainerStarted","Data":"ca7cfe2506a8f4c489a5ad16c8017bac4196765639fab1ffdf9d02444b565ee1"} Nov 01 00:50:38.002040 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.002008 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:38.004706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.004499 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 00:50:38.004706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.004556 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 00:50:38.091145 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.091003 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podStartSLOduration=2.090983556 podStartE2EDuration="2.090983556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:38.088804096 +0000 UTC m=+1628.068386457" watchObservedRunningTime="2026-11-01 00:50:38.090983556 +0000 UTC m=+1628.070565893" Nov 01 00:50:38.204841 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.204796 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718" path="/var/lib/kubelet/pods/d01c4e4e-9c3f-4b2c-9f2d-5f74782fd718/volumes" Nov 01 00:50:38.228739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.227618 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podStartSLOduration=2.227598288 podStartE2EDuration="2.227598288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:38.220680877 +0000 UTC m=+1628.200263210" watchObservedRunningTime="2026-11-01 00:50:38.227598288 +0000 UTC m=+1628.207180624" Nov 01 00:50:38.367801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.367709 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:38.371779 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.371746 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 00:50:38.372021 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:38.371995 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 00:50:39.002128 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.002090 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 00:50:39.002769 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.002739 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 00:50:39.374707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.374070 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 00:50:39.374707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.374137 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 00:50:39.504733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.502317 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm"] Nov 01 00:50:39.512675 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.511816 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.535042 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.535008 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm"] Nov 01 00:50:39.616710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.614709 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:50:39.616710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.614865 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:50:39.616710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.614874 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dm9\" (UniqueName: \"kubernetes.io/projected/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-kube-api-access-r7dm9\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.616710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.615068 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.616710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.615113 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.717023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.716934 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dm9\" (UniqueName: \"kubernetes.io/projected/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-kube-api-access-r7dm9\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.717196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.717133 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.717196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.717178 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.718391 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.718365 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.720727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.720704 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.808579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.808348 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dm9\" (UniqueName: \"kubernetes.io/projected/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-kube-api-access-r7dm9\") pod \"maas-api-revoke-sub-28354599-dsrtm\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:39.827099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:39.827063 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:40.022681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.018746 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 00:50:40.022681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.018805 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 00:50:40.112716 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.112684 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg"] Nov 01 00:50:40.129000 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.128967 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caed0494-9xfsg"] Nov 01 00:50:40.229436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.229408 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20f6454-44fb-46ea-aec7-63a349190c59" path="/var/lib/kubelet/pods/e20f6454-44fb-46ea-aec7-63a349190c59/volumes" Nov 01 00:50:40.312997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.312966 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm"] Nov 01 00:50:40.317741 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:40.317670 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3eb53b1_f81b_4dbc_9270_8a9b8e2d041f.slice/crio-c2ac304a2065f7e0c8d69489c4f5c9e790041e78192d9e4f66e1c1cb1ce66d2b WatchSource:0}: Error finding container c2ac304a2065f7e0c8d69489c4f5c9e790041e78192d9e4f66e1c1cb1ce66d2b: Status 404 returned error can't find the container with id c2ac304a2065f7e0c8d69489c4f5c9e790041e78192d9e4f66e1c1cb1ce66d2b Nov 01 00:50:40.371921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.371886 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 00:50:40.372386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:40.372257 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 00:50:41.010651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:41.009072 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 00:50:41.010651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:41.009137 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 00:50:41.038135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:41.038088 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" event={"ID":"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f","Type":"ContainerStarted","Data":"0dd5778da99d0faf2d8a5b323896ea328bcabcc26210c139fff92480a56e2a22"} Nov 01 00:50:41.038559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:41.038149 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" event={"ID":"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f","Type":"ContainerStarted","Data":"c2ac304a2065f7e0c8d69489c4f5c9e790041e78192d9e4f66e1c1cb1ce66d2b"} Nov 01 00:50:41.390735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:41.387758 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 00:50:41.390735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:41.387820 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 00:50:42.015672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:42.014743 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 00:50:42.015672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:42.014825 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 00:50:42.049504 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:42.049474 4910 generic.go:356] "Generic (PLEG): container finished" podID="d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" containerID="0dd5778da99d0faf2d8a5b323896ea328bcabcc26210c139fff92480a56e2a22" exitCode=0 Nov 01 00:50:42.049990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:42.049559 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" event={"ID":"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f","Type":"ContainerDied","Data":"0dd5778da99d0faf2d8a5b323896ea328bcabcc26210c139fff92480a56e2a22"} Nov 01 00:50:42.371705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:42.371129 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 00:50:42.371705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:42.371189 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 00:50:43.002138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.002102 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 00:50:43.002460 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.002428 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 00:50:43.129805 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:50:43.129770 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:50:43.402869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.386230 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 00:50:43.402869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.386289 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 00:50:43.404849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.404817 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:43.542208 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.542176 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-openshift-ca\") pod \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " Nov 01 00:50:43.542359 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.542222 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-kube-api-access-r7dm9\" (UniqueName: \"kubernetes.io/projected/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-kube-api-access-r7dm9\") pod \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " Nov 01 00:50:43.542411 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.542374 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-xks-ca\") pod \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\" (UID: \"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f\") " Nov 01 00:50:43.542565 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.542543 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-openshift-ca" pod "d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" (UID: "d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:43.543026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.543002 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:43.545283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.544715 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-xks-ca" pod "d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" (UID: "d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:43.545283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.544740 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-kube-api-access-r7dm9" pod "d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" (UID: "d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f"). InnerVolumeSpecName "kube-api-access-r7dm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:43.645290 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.645253 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:43.645290 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.645293 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dm9\" (UniqueName: \"kubernetes.io/projected/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f-kube-api-access-r7dm9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:43.716326 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.716275 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k"] Nov 01 00:50:43.717614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.717585 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" containerName="revoke-keys" Nov 01 00:50:43.717812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.717797 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" containerName="revoke-keys" Nov 01 00:50:43.718139 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.718123 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" containerName="revoke-keys" Nov 01 00:50:43.724521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.723648 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.739470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.739444 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k"] Nov 01 00:50:43.849193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.849109 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxr7\" (UniqueName: \"kubernetes.io/projected/3f724eb2-134b-43a5-9517-f65caa770eb1-kube-api-access-fdxr7\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.849339 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.849273 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.849339 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.849332 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.970452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.969637 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.970452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.969745 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.970452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.969942 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxr7\" (UniqueName: \"kubernetes.io/projected/3f724eb2-134b-43a5-9517-f65caa770eb1-kube-api-access-fdxr7\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.971200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.971172 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:43.973581 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:43.973558 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:44.001497 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.001462 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 01 00:50:44.001711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.001529 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 01 00:50:44.005841 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.005807 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl"] Nov 01 00:50:44.011074 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.011050 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.031447 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.031424 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxr7\" (UniqueName: \"kubernetes.io/projected/3f724eb2-134b-43a5-9517-f65caa770eb1-kube-api-access-fdxr7\") pod \"maas-api-revoke-sub-5f5d94fa-j5l8k\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:44.048885 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.048854 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:44.050463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.050436 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t"] Nov 01 00:50:44.059136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.059110 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3b9220-mq98t"] Nov 01 00:50:44.067871 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.067819 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl"] Nov 01 00:50:44.116740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.115369 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" event={"ID":"d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f","Type":"ContainerDied","Data":"c2ac304a2065f7e0c8d69489c4f5c9e790041e78192d9e4f66e1c1cb1ce66d2b"} Nov 01 00:50:44.116740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.115429 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ac304a2065f7e0c8d69489c4f5c9e790041e78192d9e4f66e1c1cb1ce66d2b" Nov 01 00:50:44.116740 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.115498 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm" Nov 01 00:50:44.174336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.173233 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kxtn\" (UniqueName: \"kubernetes.io/projected/63d021e4-e076-43a9-a60d-3d62a728f24d-kube-api-access-8kxtn\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.174336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.173385 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.174336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.173432 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.201024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.200941 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a24ae4-683d-4664-ade8-24a0cdf3832e" path="/var/lib/kubelet/pods/48a24ae4-683d-4664-ade8-24a0cdf3832e/volumes" Nov 01 00:50:44.275424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.275385 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kxtn\" (UniqueName: \"kubernetes.io/projected/63d021e4-e076-43a9-a60d-3d62a728f24d-kube-api-access-8kxtn\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.275651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.275529 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.275651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.275572 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.276442 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.276413 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.278152 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.278127 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.338845 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.338806 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kxtn\" (UniqueName: \"kubernetes.io/projected/63d021e4-e076-43a9-a60d-3d62a728f24d-kube-api-access-8kxtn\") pod \"maas-api-revoke-sub-f5f04d4b-ljqsl\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:44.365963 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.365930 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 00:50:44.366135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.365993 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 00:50:44.477705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.469940 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k"] Nov 01 00:50:44.630402 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:44.630369 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:45.059505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.059475 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl"] Nov 01 00:50:45.134427 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.134326 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" event={"ID":"3f724eb2-134b-43a5-9517-f65caa770eb1","Type":"ContainerStarted","Data":"69cf5614bee4d337b65ddfe9a5969ff5de630c9319cd9417671ea63bb928eef8"} Nov 01 00:50:45.134593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.134448 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" event={"ID":"3f724eb2-134b-43a5-9517-f65caa770eb1","Type":"ContainerStarted","Data":"f2e2948a4351ef83f269cc47742940e8a9798a8c3cd9156c4021412caa091307"} Nov 01 00:50:45.136673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.136604 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" event={"ID":"63d021e4-e076-43a9-a60d-3d62a728f24d","Type":"ContainerStarted","Data":"a853b70f19f254dbe77f04c4a707b59651423423cf3573481ebdd13856a6e478"} Nov 01 00:50:45.228339 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.227730 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:45.229028 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.229007 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:45.229787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.229764 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 00:50:45.385404 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.385260 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" podStartSLOduration=2.385239734 podStartE2EDuration="2.385239734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:45.25486143 +0000 UTC m=+1635.234443766" watchObservedRunningTime="2026-11-01 00:50:45.385239734 +0000 UTC m=+1635.364822070" Nov 01 00:50:45.537382 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.537341 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:45.537824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.537798 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:45.538527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.538503 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 00:50:45.932635 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.932548 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w"] Nov 01 00:50:45.937372 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.937352 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:45.998193 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:45.998157 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w"] Nov 01 00:50:46.009930 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.009896 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5nr\" (UniqueName: \"kubernetes.io/projected/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-kube-api-access-mx5nr\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.010075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.010037 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.010283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.010215 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.111671 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.111605 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5nr\" (UniqueName: \"kubernetes.io/projected/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-kube-api-access-mx5nr\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.111820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.111730 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.111920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.111905 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.112432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.112397 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.114673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.114648 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.149262 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.149197 4910 generic.go:356] "Generic (PLEG): container finished" podID="3f724eb2-134b-43a5-9517-f65caa770eb1" containerID="69cf5614bee4d337b65ddfe9a5969ff5de630c9319cd9417671ea63bb928eef8" exitCode=0 Nov 01 00:50:46.149438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.149282 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" event={"ID":"3f724eb2-134b-43a5-9517-f65caa770eb1","Type":"ContainerDied","Data":"69cf5614bee4d337b65ddfe9a5969ff5de630c9319cd9417671ea63bb928eef8"} Nov 01 00:50:46.151556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.151526 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" event={"ID":"63d021e4-e076-43a9-a60d-3d62a728f24d","Type":"ContainerStarted","Data":"4934af0c6fd5398fd88d9ccf2e7fcd45eedbaea8ad3ad19b9c8c78ca0b44662d"} Nov 01 00:50:46.173331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.173305 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5nr\" (UniqueName: \"kubernetes.io/projected/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-kube-api-access-mx5nr\") pod \"maas-api-revoke-sub-1cf54b5e-ckk6w\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.270586 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.270551 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:46.670435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.670341 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" podStartSLOduration=3.670326423 podStartE2EDuration="3.670326423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:46.388432535 +0000 UTC m=+1636.368014871" watchObservedRunningTime="2026-11-01 00:50:46.670326423 +0000 UTC m=+1636.649908784" Nov 01 00:50:46.674700 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:46.674666 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700bcef9_a81d_4bdb_84ae_492d2b1bb50f.slice/crio-ba5f6ec9015ac3b65a736a55429b1579c2eadb63a7810548b6950a59f61f6c85 WatchSource:0}: Error finding container ba5f6ec9015ac3b65a736a55429b1579c2eadb63a7810548b6950a59f61f6c85: Status 404 returned error can't find the container with id ba5f6ec9015ac3b65a736a55429b1579c2eadb63a7810548b6950a59f61f6c85 Nov 01 00:50:46.676978 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:46.676953 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w"] Nov 01 00:50:47.183020 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.182982 4910 generic.go:356] "Generic (PLEG): container finished" podID="63d021e4-e076-43a9-a60d-3d62a728f24d" containerID="4934af0c6fd5398fd88d9ccf2e7fcd45eedbaea8ad3ad19b9c8c78ca0b44662d" exitCode=0 Nov 01 00:50:47.183215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.183112 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" event={"ID":"63d021e4-e076-43a9-a60d-3d62a728f24d","Type":"ContainerDied","Data":"4934af0c6fd5398fd88d9ccf2e7fcd45eedbaea8ad3ad19b9c8c78ca0b44662d"} Nov 01 00:50:47.187215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.187186 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" event={"ID":"700bcef9-a81d-4bdb-84ae-492d2b1bb50f","Type":"ContainerStarted","Data":"51d884f7e85e5af09e9716e18906fa956e416acc659eeec6aea98b002bd3b292"} Nov 01 00:50:47.187341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.187230 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" event={"ID":"700bcef9-a81d-4bdb-84ae-492d2b1bb50f","Type":"ContainerStarted","Data":"ba5f6ec9015ac3b65a736a55429b1579c2eadb63a7810548b6950a59f61f6c85"} Nov 01 00:50:47.413307 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.413279 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:47.416322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.416279 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" podStartSLOduration=2.4162674060000002 podStartE2EDuration="2.416267406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:47.415179256 +0000 UTC m=+1637.394761583" watchObservedRunningTime="2026-11-01 00:50:47.416267406 +0000 UTC m=+1637.395849740" Nov 01 00:50:47.439094 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.439070 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-openshift-ca\") pod \"3f724eb2-134b-43a5-9517-f65caa770eb1\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " Nov 01 00:50:47.439225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.439138 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-xks-ca\") pod \"3f724eb2-134b-43a5-9517-f65caa770eb1\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " Nov 01 00:50:47.439279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.439258 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f724eb2-134b-43a5-9517-f65caa770eb1-kube-api-access-fdxr7\" (UniqueName: \"kubernetes.io/projected/3f724eb2-134b-43a5-9517-f65caa770eb1-kube-api-access-fdxr7\") pod \"3f724eb2-134b-43a5-9517-f65caa770eb1\" (UID: \"3f724eb2-134b-43a5-9517-f65caa770eb1\") " Nov 01 00:50:47.439466 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.439442 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-openshift-ca" pod "3f724eb2-134b-43a5-9517-f65caa770eb1" (UID: "3f724eb2-134b-43a5-9517-f65caa770eb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:47.439868 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.439848 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.441439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.441414 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-xks-ca" pod "3f724eb2-134b-43a5-9517-f65caa770eb1" (UID: "3f724eb2-134b-43a5-9517-f65caa770eb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:47.441557 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.441522 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f724eb2-134b-43a5-9517-f65caa770eb1-kube-api-access-fdxr7" pod "3f724eb2-134b-43a5-9517-f65caa770eb1" (UID: "3f724eb2-134b-43a5-9517-f65caa770eb1"). InnerVolumeSpecName "kube-api-access-fdxr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:47.541096 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.541037 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdxr7\" (UniqueName: \"kubernetes.io/projected/3f724eb2-134b-43a5-9517-f65caa770eb1-kube-api-access-fdxr7\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:47.541096 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:47.541067 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f724eb2-134b-43a5-9517-f65caa770eb1-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:48.193897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.193858 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7116-102294"] Nov 01 00:50:48.194852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.194831 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3f724eb2-134b-43a5-9517-f65caa770eb1" containerName="revoke-keys" Nov 01 00:50:48.194852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.194850 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f724eb2-134b-43a5-9517-f65caa770eb1" containerName="revoke-keys" Nov 01 00:50:48.195054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.194968 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f724eb2-134b-43a5-9517-f65caa770eb1" containerName="revoke-keys" Nov 01 00:50:48.198745 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.198720 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-102294" Nov 01 00:50:48.201317 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.201296 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" Nov 01 00:50:48.201437 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.201334 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k" event={"ID":"3f724eb2-134b-43a5-9517-f65caa770eb1","Type":"ContainerDied","Data":"f2e2948a4351ef83f269cc47742940e8a9798a8c3cd9156c4021412caa091307"} Nov 01 00:50:48.201437 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.201386 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e2948a4351ef83f269cc47742940e8a9798a8c3cd9156c4021412caa091307" Nov 01 00:50:48.203576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.203549 4910 generic.go:356] "Generic (PLEG): container finished" podID="700bcef9-a81d-4bdb-84ae-492d2b1bb50f" containerID="51d884f7e85e5af09e9716e18906fa956e416acc659eeec6aea98b002bd3b292" exitCode=0 Nov 01 00:50:48.203701 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.203616 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" event={"ID":"700bcef9-a81d-4bdb-84ae-492d2b1bb50f","Type":"ContainerDied","Data":"51d884f7e85e5af09e9716e18906fa956e416acc659eeec6aea98b002bd3b292"} Nov 01 00:50:48.243112 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.243075 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-102294"] Nov 01 00:50:48.274121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.273508 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4c5\" (UniqueName: \"kubernetes.io/projected/7a8b5af7-685f-4862-b524-e511a740cb81-kube-api-access-np4c5\") pod \"test-curl-7116-102294\" (UID: \"7a8b5af7-685f-4862-b524-e511a740cb81\") " pod="opendatahub/test-curl-7116-102294" Nov 01 00:50:48.286054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.286022 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2k2hw\"" Nov 01 00:50:48.375816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.375780 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4c5\" (UniqueName: \"kubernetes.io/projected/7a8b5af7-685f-4862-b524-e511a740cb81-kube-api-access-np4c5\") pod \"test-curl-7116-102294\" (UID: \"7a8b5af7-685f-4862-b524-e511a740cb81\") " pod="opendatahub/test-curl-7116-102294" Nov 01 00:50:48.404348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.404320 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:48.437232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.437203 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4c5\" (UniqueName: \"kubernetes.io/projected/7a8b5af7-685f-4862-b524-e511a740cb81-kube-api-access-np4c5\") pod \"test-curl-7116-102294\" (UID: \"7a8b5af7-685f-4862-b524-e511a740cb81\") " pod="opendatahub/test-curl-7116-102294" Nov 01 00:50:48.476986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.476871 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d021e4-e076-43a9-a60d-3d62a728f24d-kube-api-access-8kxtn\" (UniqueName: \"kubernetes.io/projected/63d021e4-e076-43a9-a60d-3d62a728f24d-kube-api-access-8kxtn\") pod \"63d021e4-e076-43a9-a60d-3d62a728f24d\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " Nov 01 00:50:48.476986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.476913 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-openshift-ca\") pod \"63d021e4-e076-43a9-a60d-3d62a728f24d\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " Nov 01 00:50:48.476986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.476954 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-xks-ca\") pod \"63d021e4-e076-43a9-a60d-3d62a728f24d\" (UID: \"63d021e4-e076-43a9-a60d-3d62a728f24d\") " Nov 01 00:50:48.477380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.477353 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-openshift-ca" pod "63d021e4-e076-43a9-a60d-3d62a728f24d" (UID: "63d021e4-e076-43a9-a60d-3d62a728f24d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:48.477617 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.477576 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:48.479137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.479112 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-xks-ca" pod "63d021e4-e076-43a9-a60d-3d62a728f24d" (UID: "63d021e4-e076-43a9-a60d-3d62a728f24d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:48.479244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.479210 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d021e4-e076-43a9-a60d-3d62a728f24d-kube-api-access-8kxtn" pod "63d021e4-e076-43a9-a60d-3d62a728f24d" (UID: "63d021e4-e076-43a9-a60d-3d62a728f24d"). InnerVolumeSpecName "kube-api-access-8kxtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:48.516983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.516944 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-102294" Nov 01 00:50:48.579009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.578979 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d021e4-e076-43a9-a60d-3d62a728f24d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:48.579124 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.579014 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kxtn\" (UniqueName: \"kubernetes.io/projected/63d021e4-e076-43a9-a60d-3d62a728f24d-kube-api-access-8kxtn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:48.898554 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.898516 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-102294"] Nov 01 00:50:48.902775 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:50:48.902738 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a8b5af7_685f_4862_b524_e511a740cb81.slice/crio-e9d24b6ebe57e1324075c24e8e727e3fdb7c727d6f41f69d936025e0c7485b58 WatchSource:0}: Error finding container e9d24b6ebe57e1324075c24e8e727e3fdb7c727d6f41f69d936025e0c7485b58: Status 404 returned error can't find the container with id e9d24b6ebe57e1324075c24e8e727e3fdb7c727d6f41f69d936025e0c7485b58 Nov 01 00:50:48.904575 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:48.904559 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 00:50:49.214122 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.214033 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" Nov 01 00:50:49.214122 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.214087 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl" event={"ID":"63d021e4-e076-43a9-a60d-3d62a728f24d","Type":"ContainerDied","Data":"a853b70f19f254dbe77f04c4a707b59651423423cf3573481ebdd13856a6e478"} Nov 01 00:50:49.214349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.214132 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a853b70f19f254dbe77f04c4a707b59651423423cf3573481ebdd13856a6e478" Nov 01 00:50:49.215673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.215648 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-102294" event={"ID":"7a8b5af7-685f-4862-b524-e511a740cb81","Type":"ContainerStarted","Data":"e9d24b6ebe57e1324075c24e8e727e3fdb7c727d6f41f69d936025e0c7485b58"} Nov 01 00:50:49.385077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.385045 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:49.494006 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.493905 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-xks-ca\") pod \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " Nov 01 00:50:49.494006 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.494001 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-openshift-ca\") pod \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " Nov 01 00:50:49.494556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.494224 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-kube-api-access-mx5nr\" (UniqueName: \"kubernetes.io/projected/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-kube-api-access-mx5nr\") pod \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\" (UID: \"700bcef9-a81d-4bdb-84ae-492d2b1bb50f\") " Nov 01 00:50:49.494556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.494380 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-openshift-ca" pod "700bcef9-a81d-4bdb-84ae-492d2b1bb50f" (UID: "700bcef9-a81d-4bdb-84ae-492d2b1bb50f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:50:49.494745 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.494722 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:49.496359 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.496325 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-xks-ca" pod "700bcef9-a81d-4bdb-84ae-492d2b1bb50f" (UID: "700bcef9-a81d-4bdb-84ae-492d2b1bb50f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:50:49.496490 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.496429 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-kube-api-access-mx5nr" pod "700bcef9-a81d-4bdb-84ae-492d2b1bb50f" (UID: "700bcef9-a81d-4bdb-84ae-492d2b1bb50f"). InnerVolumeSpecName "kube-api-access-mx5nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:50:49.596232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.596192 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5nr\" (UniqueName: \"kubernetes.io/projected/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-kube-api-access-mx5nr\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:49.596232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:49.596227 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700bcef9-a81d-4bdb-84ae-492d2b1bb50f-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:50:50.055028 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:50.054996 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq"] Nov 01 00:50:50.063489 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:50.063456 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-382b03ad-fwchq"] Nov 01 00:50:50.196688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:50.196610 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6" path="/var/lib/kubelet/pods/1f05ff2e-a2a2-4d29-8910-c1c1a7bba0c6/volumes" Nov 01 00:50:50.226292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:50.226252 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" event={"ID":"700bcef9-a81d-4bdb-84ae-492d2b1bb50f","Type":"ContainerDied","Data":"ba5f6ec9015ac3b65a736a55429b1579c2eadb63a7810548b6950a59f61f6c85"} Nov 01 00:50:50.226292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:50.226281 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w" Nov 01 00:50:50.226497 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:50.226312 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba5f6ec9015ac3b65a736a55429b1579c2eadb63a7810548b6950a59f61f6c85" Nov 01 00:50:52.054185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:52.054150 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6"] Nov 01 00:50:52.102553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:52.102516 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76554d1a-gl9v6"] Nov 01 00:50:52.194334 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:52.194294 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b13b468-aca8-4367-bdc3-f109f8942aea" path="/var/lib/kubelet/pods/2b13b468-aca8-4367-bdc3-f109f8942aea/volumes" Nov 01 00:50:52.253588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:52.253555 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-102294" event={"ID":"7a8b5af7-685f-4862-b524-e511a740cb81","Type":"ContainerStarted","Data":"7227751ab23047152b778b53a36f2f0064a51ad6b0f2ec45d4756d6e03d466c1"} Nov 01 00:50:52.367663 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:52.367527 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7116-102294" podStartSLOduration=1.563238486 podStartE2EDuration="4.367512667s" totalImagesPullingTime="2.804274181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:50:52.366850408 +0000 UTC m=+1642.346432744" watchObservedRunningTime="2026-11-01 00:50:52.367512667 +0000 UTC m=+1642.347094999" Nov 01 00:50:53.900801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:53.899203 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7116-102294"] Nov 01 00:50:53.913943 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:53.908116 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7116-102294"] Nov 01 00:50:54.476650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.471505 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7116-a47641"] Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501501 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" containerName="test-curl-7116-102294" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501533 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" containerName="test-curl-7116-102294" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501592 4910 cpu_manager.go:414] "containerMap: removing container" podUID="63d021e4-e076-43a9-a60d-3d62a728f24d" containerName="revoke-keys" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501600 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d021e4-e076-43a9-a60d-3d62a728f24d" containerName="revoke-keys" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501617 4910 cpu_manager.go:414] "containerMap: removing container" podUID="700bcef9-a81d-4bdb-84ae-492d2b1bb50f" containerName="revoke-keys" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501642 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700bcef9-a81d-4bdb-84ae-492d2b1bb50f" containerName="revoke-keys" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501839 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" containerName="test-curl-7116-102294" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501858 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d021e4-e076-43a9-a60d-3d62a728f24d" containerName="revoke-keys" Nov 01 00:50:54.504709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.501868 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="700bcef9-a81d-4bdb-84ae-492d2b1bb50f" containerName="revoke-keys" Nov 01 00:50:54.509462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.509402 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7116-102294" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" containerName="test-curl-7116-102294" containerID="cri-o://7227751ab23047152b778b53a36f2f0064a51ad6b0f2ec45d4756d6e03d466c1" gracePeriod=2 Nov 01 00:50:54.532051 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.509802 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-a47641"] Nov 01 00:50:54.533453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.509872 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-a47641" Nov 01 00:50:54.594962 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.594927 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm278\" (UniqueName: \"kubernetes.io/projected/50023407-9ad3-4d61-a76e-c05347cceadf-kube-api-access-nm278\") pod \"test-curl-7116-a47641\" (UID: \"50023407-9ad3-4d61-a76e-c05347cceadf\") " pod="opendatahub/test-curl-7116-a47641" Nov 01 00:50:54.708616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.708570 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm278\" (UniqueName: \"kubernetes.io/projected/50023407-9ad3-4d61-a76e-c05347cceadf-kube-api-access-nm278\") pod \"test-curl-7116-a47641\" (UID: \"50023407-9ad3-4d61-a76e-c05347cceadf\") " pod="opendatahub/test-curl-7116-a47641" Nov 01 00:50:54.867647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.866459 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm278\" (UniqueName: \"kubernetes.io/projected/50023407-9ad3-4d61-a76e-c05347cceadf-kube-api-access-nm278\") pod \"test-curl-7116-a47641\" (UID: \"50023407-9ad3-4d61-a76e-c05347cceadf\") " pod="opendatahub/test-curl-7116-a47641" Nov 01 00:50:54.961649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:54.960865 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-a47641" Nov 01 00:50:55.096678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.093184 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh"] Nov 01 00:50:55.287700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.286195 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:55.362870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.362819 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.413953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.412607 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:55.442650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.438758 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:55.457571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.448749 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:50:55.462412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.461837 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjl8x\" (UniqueName: \"kubernetes.io/projected/529bb719-82c9-48e9-9845-dfe955e965c2-kube-api-access-tjl8x\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:55.462412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.462051 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:55.535877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.526766 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:50:55.535877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.526847 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:50:55.580445 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.552490 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:50:55.610975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.609925 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjl8x\" (UniqueName: \"kubernetes.io/projected/529bb719-82c9-48e9-9845-dfe955e965c2-kube-api-access-tjl8x\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:55.610975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.610047 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:55.610975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.610208 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:55.776813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:55.654703 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:56.074647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.074166 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 01 00:50:56.120296 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.085147 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjl8x\" (UniqueName: \"kubernetes.io/projected/529bb719-82c9-48e9-9845-dfe955e965c2-kube-api-access-tjl8x\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:56.210997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.123033 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c89ea6-524mh\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:56.210997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.137237 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:50:56.210997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.178742 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh"] Nov 01 00:50:56.374919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.326372 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.432392 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.422175 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.476199 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.471873 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 00:50:56.617560 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.584945 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:50:56.649592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.647743 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.751651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.725753 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.823343 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.823289 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:56.823735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.823689 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:56.926706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.925161 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:50:56.926706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.925354 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:50:56.926706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.925416 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:50:56.926706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:56.925454 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:50:57.143380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.143338 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 01 00:50:57.146736 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.146678 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:50:57.374653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.369953 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.374653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.370437 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 00:50:57.374653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.370853 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:50:57.452663 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.431737 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:50:57.551760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.550910 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.586240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.586194 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:57.634003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.633971 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-a47641"] Nov 01 00:50:57.648011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.647969 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:50:57.700582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.700531 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-a47641" event={"ID":"50023407-9ad3-4d61-a76e-c05347cceadf","Type":"ContainerStarted","Data":"8cdbe1fe8f175506e2ea5f47786d839ce85c46775610b76265fdbdb5217dcf34"} Nov 01 00:50:57.794658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.794595 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:50:57.822891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.796256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:50:57.824185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.824158 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:50:57.932042 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:57.931941 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.129708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.116599 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:58.129708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.116680 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.436705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.410572 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.445879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.445831 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:50:58.593653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.574330 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.593653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.592754 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:50:58.636653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.610484 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:50:58.641716 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.640275 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:58.802351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.802303 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:50:58.812533 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.803871 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:50:59.024526 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:58.956809 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 01 00:50:59.175958 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.175831 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:59.176381 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.176339 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.316471 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.316431 4910 generic.go:356] "Generic (PLEG): container finished" podID="7a8b5af7-685f-4862-b524-e511a740cb81" containerID="7227751ab23047152b778b53a36f2f0064a51ad6b0f2ec45d4756d6e03d466c1" exitCode=137 Nov 01 00:50:59.526651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.519766 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.602500 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.601230 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.649863 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.649806 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.682137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.682080 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.684501 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.684466 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:50:59.705262 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.705216 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.846221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.846182 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:50:59.925754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.925711 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh"] Nov 01 00:50:59.971665 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:50:59.957456 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:51:00.642255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.329189 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:00.642848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.642813 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.642987 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.417856 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.652699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.403573 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.656666 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.584089 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.661120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.602812 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.684651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.661375 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpk5m\" (UniqueName: \"kubernetes.io/projected/c5041663-87fc-4806-83d0-185c303afe2b-kube-api-access-zpk5m\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.685259 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.685223 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.685536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.601774 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:51:00.686842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.654973 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.686842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.655009 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:00.687151 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.655089 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:00.687302 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.687281 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.687455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.687143 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:00.810139 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.810101 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.810422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.810400 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpk5m\" (UniqueName: \"kubernetes.io/projected/c5041663-87fc-4806-83d0-185c303afe2b-kube-api-access-zpk5m\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.810610 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.810593 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.811874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.811850 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:00.832648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.821068 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:51:01.081900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.081869 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:51:01.082190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.082160 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh"] Nov 01 00:51:01.082372 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:00.872927 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.095347 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.012273 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-102294" Nov 01 00:51:01.095663 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.085186 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d24b6ebe57e1324075c24e8e727e3fdb7c727d6f41f69d936025e0c7485b58" Nov 01 00:51:01.143122 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.143085 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:01.143454 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.143402 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.171116 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.171019 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l"] Nov 01 00:51:01.205661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.202388 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8b5af7-685f-4862-b524-e511a740cb81-kube-api-access-np4c5\" (UniqueName: \"kubernetes.io/projected/7a8b5af7-685f-4862-b524-e511a740cb81-kube-api-access-np4c5\") pod \"7a8b5af7-685f-4862-b524-e511a740cb81\" (UID: \"7a8b5af7-685f-4862-b524-e511a740cb81\") " Nov 01 00:51:01.239416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.239370 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpk5m\" (UniqueName: \"kubernetes.io/projected/c5041663-87fc-4806-83d0-185c303afe2b-kube-api-access-zpk5m\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:01.267186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.267151 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:01.267478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.267449 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.325861 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:01.325813 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-102294\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" pod="opendatahub/test-curl-7116-102294" Nov 01 00:51:01.346324 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.339372 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:01.346651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.346575 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.417073 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.416988 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf9ebf8-vxzvh\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:01.489752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.489701 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:01.505742 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.505647 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l"] Nov 01 00:51:01.530292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.530262 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.579864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.576671 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.579864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.576873 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gpj\" (UniqueName: \"kubernetes.io/projected/34425d1e-8f64-4332-b32f-85a88b8bb881-kube-api-access-w6gpj\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.579864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.576943 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.663873 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.663818 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:51:01.705907 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.705845 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:01.706803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.706773 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:01.707340 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.707309 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.707771 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.707749 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 01 00:51:01.711895 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.711850 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:01.766706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.752811 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.766706 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:01.757198 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-102294\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" pod="opendatahub/test-curl-7116-102294" Nov 01 00:51:01.766706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.761325 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:01.766706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.763832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gpj\" (UniqueName: \"kubernetes.io/projected/34425d1e-8f64-4332-b32f-85a88b8bb881-kube-api-access-w6gpj\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.766706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.763907 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.766706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.764051 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.766706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.765964 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:01.955209 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:01.955167 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8b5af7-685f-4862-b524-e511a740cb81-kube-api-access-np4c5" pod "7a8b5af7-685f-4862-b524-e511a740cb81" (UID: "7a8b5af7-685f-4862-b524-e511a740cb81"). InnerVolumeSpecName "kube-api-access-np4c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:02.348647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.324834 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4c5\" (UniqueName: \"kubernetes.io/projected/7a8b5af7-685f-4862-b524-e511a740cb81-kube-api-access-np4c5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:02.348897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.342363 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:02.381453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.381411 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:51:02.448768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.443277 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gpj\" (UniqueName: \"kubernetes.io/projected/34425d1e-8f64-4332-b32f-85a88b8bb881-kube-api-access-w6gpj\") pod \"maas-api-revoke-sub-72be219b-p528l\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:02.678656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.666910 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.728036 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.727921 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:02.739163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.728820 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:02.823695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.774464 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.823695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.796016 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:02.867232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.761709 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:02.970664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.953375 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:02.976650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:02.972733 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:03.533650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:03.524336 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:03.829149 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:03.829100 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:04.139645 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:03.997806 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" path="/var/lib/kubelet/pods/7a8b5af7-685f-4862-b524-e511a740cb81/volumes" Nov 01 00:51:04.142822 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:04.141358 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.792s" Nov 01 00:51:04.169275 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.169246 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh"] Nov 01 00:51:04.175642 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.173414 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:51:04.175642 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.173518 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:51:04.175642 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.155925 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:51:04.181989 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.105121 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:04.230280 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.153247 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:04.249491 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.244810 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:51:04.249491 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.244885 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:51:04.347660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.272004 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:51:04.347660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.347406 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:51:04.374018 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.373989 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-102294" Nov 01 00:51:04.374247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.374032 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" event={"ID":"529bb719-82c9-48e9-9845-dfe955e965c2","Type":"ContainerStarted","Data":"d35a8cee0778a5a7e5a6768f9df1fd78d69342aae8432190e76b5c246605cac6"} Nov 01 00:51:04.401254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.401224 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:51:04.564644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.564603 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh"] Nov 01 00:51:04.597604 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:51:04.597565 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34425d1e_8f64_4332_b32f_85a88b8bb881.slice/crio-296c6409ca621587c41f9d25109475444f0fc1d91c16bbff9fc983bb8de4c03c WatchSource:0}: Error finding container 296c6409ca621587c41f9d25109475444f0fc1d91c16bbff9fc983bb8de4c03c: Status 404 returned error can't find the container with id 296c6409ca621587c41f9d25109475444f0fc1d91c16bbff9fc983bb8de4c03c Nov 01 00:51:04.605089 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:04.605052 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l"] Nov 01 00:51:05.094705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.094655 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" podStartSLOduration=10.094640363 podStartE2EDuration="10.094640363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:05.091668436 +0000 UTC m=+1655.071250771" watchObservedRunningTime="2026-11-01 00:51:05.094640363 +0000 UTC m=+1655.074222696" Nov 01 00:51:05.142952 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:05.142866 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-102294\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" pod="opendatahub/test-curl-7116-102294" Nov 01 00:51:05.396289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.396198 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" event={"ID":"34425d1e-8f64-4332-b32f-85a88b8bb881","Type":"ContainerStarted","Data":"6a85e0ca8f99d8e6adbe2fed5ee0334e87666849beafe5b5a693afc37c682f49"} Nov 01 00:51:05.396289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.396263 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" event={"ID":"34425d1e-8f64-4332-b32f-85a88b8bb881","Type":"ContainerStarted","Data":"296c6409ca621587c41f9d25109475444f0fc1d91c16bbff9fc983bb8de4c03c"} Nov 01 00:51:05.398225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.398196 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-a47641" event={"ID":"50023407-9ad3-4d61-a76e-c05347cceadf","Type":"ContainerStarted","Data":"886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8"} Nov 01 00:51:05.399972 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.399948 4910 generic.go:356] "Generic (PLEG): container finished" podID="529bb719-82c9-48e9-9845-dfe955e965c2" containerID="8643973d966c3a4e73f604d6e1686d65ea80c001b92f498e085c33b9b14122a6" exitCode=0 Nov 01 00:51:05.400101 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.400026 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" event={"ID":"529bb719-82c9-48e9-9845-dfe955e965c2","Type":"ContainerDied","Data":"8643973d966c3a4e73f604d6e1686d65ea80c001b92f498e085c33b9b14122a6"} Nov 01 00:51:05.401724 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.401702 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" event={"ID":"c5041663-87fc-4806-83d0-185c303afe2b","Type":"ContainerStarted","Data":"55febcb4e354476b5891fd920f7a3c7804ff96b4f93c2e4ec6a21d99afc0ffd4"} Nov 01 00:51:05.401813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.401756 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" event={"ID":"c5041663-87fc-4806-83d0-185c303afe2b","Type":"ContainerStarted","Data":"b1ed03eed387d1e7dc938da4740ba7f7da587a26702bbb589ff80288711f36c9"} Nov 01 00:51:05.515180 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.515124 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" podStartSLOduration=7.515110943 podStartE2EDuration="7.515110943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:05.511335495 +0000 UTC m=+1655.490917830" watchObservedRunningTime="2026-11-01 00:51:05.515110943 +0000 UTC m=+1655.494693325" Nov 01 00:51:05.562585 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:05.562547 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-102294\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" pod="opendatahub/test-curl-7116-102294" Nov 01 00:51:05.683373 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.683261 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" podStartSLOduration=8.683239889 podStartE2EDuration="8.683239889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:05.677721658 +0000 UTC m=+1655.657303994" watchObservedRunningTime="2026-11-01 00:51:05.683239889 +0000 UTC m=+1655.662822225" Nov 01 00:51:05.848257 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:05.848223 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-102294\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="7a8b5af7-685f-4862-b524-e511a740cb81" pod="opendatahub/test-curl-7116-102294" Nov 01 00:51:05.963127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:05.963027 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7116-a47641" podStartSLOduration=11.963011037 podStartE2EDuration="11.963011037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:05.961426619 +0000 UTC m=+1655.941008955" watchObservedRunningTime="2026-11-01 00:51:05.963011037 +0000 UTC m=+1655.942593372" Nov 01 00:51:06.414688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.414656 4910 generic.go:356] "Generic (PLEG): container finished" podID="34425d1e-8f64-4332-b32f-85a88b8bb881" containerID="6a85e0ca8f99d8e6adbe2fed5ee0334e87666849beafe5b5a693afc37c682f49" exitCode=0 Nov 01 00:51:06.415166 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.414757 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" event={"ID":"34425d1e-8f64-4332-b32f-85a88b8bb881","Type":"ContainerDied","Data":"6a85e0ca8f99d8e6adbe2fed5ee0334e87666849beafe5b5a693afc37c682f49"} Nov 01 00:51:06.417011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.416972 4910 generic.go:356] "Generic (PLEG): container finished" podID="c5041663-87fc-4806-83d0-185c303afe2b" containerID="55febcb4e354476b5891fd920f7a3c7804ff96b4f93c2e4ec6a21d99afc0ffd4" exitCode=0 Nov 01 00:51:06.417141 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.417043 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" event={"ID":"c5041663-87fc-4806-83d0-185c303afe2b","Type":"ContainerDied","Data":"55febcb4e354476b5891fd920f7a3c7804ff96b4f93c2e4ec6a21d99afc0ffd4"} Nov 01 00:51:06.593830 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.593805 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:51:06.601890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.601859 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-xks-ca\") pod \"529bb719-82c9-48e9-9845-dfe955e965c2\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " Nov 01 00:51:06.602025 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.601906 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-openshift-ca\") pod \"529bb719-82c9-48e9-9845-dfe955e965c2\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " Nov 01 00:51:06.602154 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.602136 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529bb719-82c9-48e9-9845-dfe955e965c2-kube-api-access-tjl8x\" (UniqueName: \"kubernetes.io/projected/529bb719-82c9-48e9-9845-dfe955e965c2-kube-api-access-tjl8x\") pod \"529bb719-82c9-48e9-9845-dfe955e965c2\" (UID: \"529bb719-82c9-48e9-9845-dfe955e965c2\") " Nov 01 00:51:06.602315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.602294 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-openshift-ca" pod "529bb719-82c9-48e9-9845-dfe955e965c2" (UID: "529bb719-82c9-48e9-9845-dfe955e965c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:06.602754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.602692 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:06.604088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.604066 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-xks-ca" pod "529bb719-82c9-48e9-9845-dfe955e965c2" (UID: "529bb719-82c9-48e9-9845-dfe955e965c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:51:06.604229 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.604215 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529bb719-82c9-48e9-9845-dfe955e965c2-kube-api-access-tjl8x" pod "529bb719-82c9-48e9-9845-dfe955e965c2" (UID: "529bb719-82c9-48e9-9845-dfe955e965c2"). InnerVolumeSpecName "kube-api-access-tjl8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:06.704206 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.704126 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjl8x\" (UniqueName: \"kubernetes.io/projected/529bb719-82c9-48e9-9845-dfe955e965c2-kube-api-access-tjl8x\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:06.704206 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:06.704165 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529bb719-82c9-48e9-9845-dfe955e965c2-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:07.427443 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.427413 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" Nov 01 00:51:07.427865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.427459 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh" event={"ID":"529bb719-82c9-48e9-9845-dfe955e965c2","Type":"ContainerDied","Data":"d35a8cee0778a5a7e5a6768f9df1fd78d69342aae8432190e76b5c246605cac6"} Nov 01 00:51:07.427865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.427507 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35a8cee0778a5a7e5a6768f9df1fd78d69342aae8432190e76b5c246605cac6" Nov 01 00:51:07.548828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.547969 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7116-a47641"] Nov 01 00:51:07.548828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.548203 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7116-a47641" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" containerName="test-curl-7116-a47641" containerID="cri-o://886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8" gracePeriod=2 Nov 01 00:51:07.596434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.596400 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7116-a47641"] Nov 01 00:51:07.599777 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:07.599741 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-a47641\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" pod="opendatahub/test-curl-7116-a47641" Nov 01 00:51:07.666137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.666109 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:07.668892 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.668864 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:07.719154 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.719047 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-xks-ca\") pod \"34425d1e-8f64-4332-b32f-85a88b8bb881\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " Nov 01 00:51:07.719154 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.719115 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-openshift-ca\") pod \"c5041663-87fc-4806-83d0-185c303afe2b\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " Nov 01 00:51:07.719400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.719177 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5041663-87fc-4806-83d0-185c303afe2b-kube-api-access-zpk5m\" (UniqueName: \"kubernetes.io/projected/c5041663-87fc-4806-83d0-185c303afe2b-kube-api-access-zpk5m\") pod \"c5041663-87fc-4806-83d0-185c303afe2b\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " Nov 01 00:51:07.719400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.719210 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-openshift-ca\") pod \"34425d1e-8f64-4332-b32f-85a88b8bb881\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " Nov 01 00:51:07.719400 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.719262 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34425d1e-8f64-4332-b32f-85a88b8bb881-kube-api-access-w6gpj\" (UniqueName: \"kubernetes.io/projected/34425d1e-8f64-4332-b32f-85a88b8bb881-kube-api-access-w6gpj\") pod \"34425d1e-8f64-4332-b32f-85a88b8bb881\" (UID: \"34425d1e-8f64-4332-b32f-85a88b8bb881\") " Nov 01 00:51:07.719576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.719562 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-xks-ca\") pod \"c5041663-87fc-4806-83d0-185c303afe2b\" (UID: \"c5041663-87fc-4806-83d0-185c303afe2b\") " Nov 01 00:51:07.720670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.719922 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-openshift-ca" pod "c5041663-87fc-4806-83d0-185c303afe2b" (UID: "c5041663-87fc-4806-83d0-185c303afe2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:07.720670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.720607 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:07.720950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.720919 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-openshift-ca" pod "34425d1e-8f64-4332-b32f-85a88b8bb881" (UID: "34425d1e-8f64-4332-b32f-85a88b8bb881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:07.721926 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.721900 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5041663-87fc-4806-83d0-185c303afe2b-kube-api-access-zpk5m" pod "c5041663-87fc-4806-83d0-185c303afe2b" (UID: "c5041663-87fc-4806-83d0-185c303afe2b"). InnerVolumeSpecName "kube-api-access-zpk5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:07.722034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.722008 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-xks-ca" pod "34425d1e-8f64-4332-b32f-85a88b8bb881" (UID: "34425d1e-8f64-4332-b32f-85a88b8bb881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:51:07.722402 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.722385 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-xks-ca" pod "c5041663-87fc-4806-83d0-185c303afe2b" (UID: "c5041663-87fc-4806-83d0-185c303afe2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:51:07.722830 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.722806 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34425d1e-8f64-4332-b32f-85a88b8bb881-kube-api-access-w6gpj" pod "34425d1e-8f64-4332-b32f-85a88b8bb881" (UID: "34425d1e-8f64-4332-b32f-85a88b8bb881"). InnerVolumeSpecName "kube-api-access-w6gpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:07.822875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.822829 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5041663-87fc-4806-83d0-185c303afe2b-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:07.822875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.822873 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:07.823085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.822887 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpk5m\" (UniqueName: \"kubernetes.io/projected/c5041663-87fc-4806-83d0-185c303afe2b-kube-api-access-zpk5m\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:07.823085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.822905 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34425d1e-8f64-4332-b32f-85a88b8bb881-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:07.823085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:07.822919 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gpj\" (UniqueName: \"kubernetes.io/projected/34425d1e-8f64-4332-b32f-85a88b8bb881-kube-api-access-w6gpj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:07.836470 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:07.836429 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-a47641\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" pod="opendatahub/test-curl-7116-a47641" Nov 01 00:51:08.029515 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:08.029476 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-a47641\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" pod="opendatahub/test-curl-7116-a47641" Nov 01 00:51:08.438308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:08.438205 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" event={"ID":"c5041663-87fc-4806-83d0-185c303afe2b","Type":"ContainerDied","Data":"b1ed03eed387d1e7dc938da4740ba7f7da587a26702bbb589ff80288711f36c9"} Nov 01 00:51:08.438308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:08.438280 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ed03eed387d1e7dc938da4740ba7f7da587a26702bbb589ff80288711f36c9" Nov 01 00:51:08.438308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:08.438225 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh" Nov 01 00:51:08.439985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:08.439952 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" Nov 01 00:51:08.440156 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:08.440127 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l" event={"ID":"34425d1e-8f64-4332-b32f-85a88b8bb881","Type":"ContainerDied","Data":"296c6409ca621587c41f9d25109475444f0fc1d91c16bbff9fc983bb8de4c03c"} Nov 01 00:51:08.440270 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:08.440175 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="296c6409ca621587c41f9d25109475444f0fc1d91c16bbff9fc983bb8de4c03c" Nov 01 00:51:09.586711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.586680 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5"] Nov 01 00:51:09.587664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587642 4910 cpu_manager.go:414] "containerMap: removing container" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" containerName="test-curl-7116-a47641" Nov 01 00:51:09.587777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587665 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" containerName="test-curl-7116-a47641" Nov 01 00:51:09.587777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587679 4910 cpu_manager.go:414] "containerMap: removing container" podUID="34425d1e-8f64-4332-b32f-85a88b8bb881" containerName="revoke-keys" Nov 01 00:51:09.587777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587687 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34425d1e-8f64-4332-b32f-85a88b8bb881" containerName="revoke-keys" Nov 01 00:51:09.587777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587732 4910 cpu_manager.go:414] "containerMap: removing container" podUID="529bb719-82c9-48e9-9845-dfe955e965c2" containerName="revoke-keys" Nov 01 00:51:09.587777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587740 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529bb719-82c9-48e9-9845-dfe955e965c2" containerName="revoke-keys" Nov 01 00:51:09.587777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587759 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c5041663-87fc-4806-83d0-185c303afe2b" containerName="revoke-keys" Nov 01 00:51:09.587777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587765 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5041663-87fc-4806-83d0-185c303afe2b" containerName="revoke-keys" Nov 01 00:51:09.588125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587964 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="529bb719-82c9-48e9-9845-dfe955e965c2" containerName="revoke-keys" Nov 01 00:51:09.588125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587980 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5041663-87fc-4806-83d0-185c303afe2b" containerName="revoke-keys" Nov 01 00:51:09.588125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.587999 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" containerName="test-curl-7116-a47641" Nov 01 00:51:09.588125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.588011 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="34425d1e-8f64-4332-b32f-85a88b8bb881" containerName="revoke-keys" Nov 01 00:51:09.593562 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.593536 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668390 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668335 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-certs\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668424 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istiod-ca-cert\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668489 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-token\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668527 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-data\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668601 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-socket\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668651 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-podinfo\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668745 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmgf\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-kube-api-access-zhmgf\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.668791 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668783 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-envoy\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.669034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.668813 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-credential-socket\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.671949 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.671921 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5"] Nov 01 00:51:09.695393 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.695358 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8a74f100-openshift-default-dockercfg-45tm4\"" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783209 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmgf\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-kube-api-access-zhmgf\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783267 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-envoy\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783298 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-credential-socket\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783682 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-certs\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783765 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istiod-ca-cert\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783836 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-token\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783876 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-data\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783962 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-socket\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.784678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.783994 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-podinfo\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.786272 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.785831 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-certs\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.789414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.787104 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-credential-socket\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.789414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.788957 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-data\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.789414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.789196 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-socket\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.789414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.789373 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istiod-ca-cert\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.805019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.804942 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-envoy\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.805435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.805411 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-podinfo\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.873076 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.872012 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmgf\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-kube-api-access-zhmgf\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.889105 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.875724 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-token\") pod \"e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:09.927695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.927493 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-a47641" Nov 01 00:51:09.940864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:09.940826 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:10.000167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.000117 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50023407-9ad3-4d61-a76e-c05347cceadf-kube-api-access-nm278\" (UniqueName: \"kubernetes.io/projected/50023407-9ad3-4d61-a76e-c05347cceadf-kube-api-access-nm278\") pod \"50023407-9ad3-4d61-a76e-c05347cceadf\" (UID: \"50023407-9ad3-4d61-a76e-c05347cceadf\") " Nov 01 00:51:10.004850 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.004685 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50023407-9ad3-4d61-a76e-c05347cceadf-kube-api-access-nm278" pod "50023407-9ad3-4d61-a76e-c05347cceadf" (UID: "50023407-9ad3-4d61-a76e-c05347cceadf"). InnerVolumeSpecName "kube-api-access-nm278". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:10.005173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.005154 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm278\" (UniqueName: \"kubernetes.io/projected/50023407-9ad3-4d61-a76e-c05347cceadf-kube-api-access-nm278\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:10.216700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.216567 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50023407-9ad3-4d61-a76e-c05347cceadf" path="/var/lib/kubelet/pods/50023407-9ad3-4d61-a76e-c05347cceadf/volumes" Nov 01 00:51:10.413804 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:51:10.413770 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ddff29_fa3e_49e0_b6f4_71c62187e64f.slice/crio-b328090b230fe93f71be83b51ba9efc4e37c2936ef1df4a5736df7a12ac9a7d0 WatchSource:0}: Error finding container b328090b230fe93f71be83b51ba9efc4e37c2936ef1df4a5736df7a12ac9a7d0: Status 404 returned error can't find the container with id b328090b230fe93f71be83b51ba9efc4e37c2936ef1df4a5736df7a12ac9a7d0 Nov 01 00:51:10.417830 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.415312 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5"] Nov 01 00:51:10.423165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.423122 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:10.423269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.423222 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:10.423308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.423263 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:10.466491 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.466453 4910 generic.go:356] "Generic (PLEG): container finished" podID="50023407-9ad3-4d61-a76e-c05347cceadf" containerID="886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8" exitCode=137 Nov 01 00:51:10.466682 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.466534 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-a47641" Nov 01 00:51:10.466788 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.466700 4910 scope.go:122] "RemoveContainer" containerID="886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8" Nov 01 00:51:10.469513 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.469471 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" event={"ID":"69ddff29-fa3e-49e0-b6f4-71c62187e64f","Type":"ContainerStarted","Data":"b328090b230fe93f71be83b51ba9efc4e37c2936ef1df4a5736df7a12ac9a7d0"} Nov 01 00:51:10.490604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.490574 4910 scope.go:122] "RemoveContainer" containerID="886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8" Nov 01 00:51:10.491022 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:10.490995 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8\": container with ID starting with 886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8 not found: ID does not exist" containerID="886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8" Nov 01 00:51:10.491185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.491157 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8"} err="failed to get container status \"886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8\": rpc error: code = NotFound desc = could not find container \"886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8\": container with ID starting with 886d1d5dfd601697f5f52f64527e63423cb270e32033a109191b07af5a2ed0e8 not found: ID does not exist" Nov 01 00:51:10.647052 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.647009 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2"] Nov 01 00:51:10.652966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.652941 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.706084 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.706049 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:51:10.706276 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.706119 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:51:10.706988 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.706962 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2"] Nov 01 00:51:10.719717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.717440 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.719717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.717674 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.719717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.717812 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgl42\" (UniqueName: \"kubernetes.io/projected/6d8804ef-0225-471a-8429-43442211dcdb-kube-api-access-jgl42\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.819121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.819082 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.819482 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.819457 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl42\" (UniqueName: \"kubernetes.io/projected/6d8804ef-0225-471a-8429-43442211dcdb-kube-api-access-jgl42\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.819673 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.819545 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.819971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.819945 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.822815 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.822790 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.881441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.881413 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl42\" (UniqueName: \"kubernetes.io/projected/6d8804ef-0225-471a-8429-43442211dcdb-kube-api-access-jgl42\") pod \"maas-api-revoke-sub-f4b68245-82wr2\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:10.969816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:10.969721 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:11.365787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:11.365756 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2"] Nov 01 00:51:11.371517 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:51:11.371485 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8804ef_0225_471a_8429_43442211dcdb.slice/crio-93aa7f92b9bcc08c666ee181695d56b209fb65b67c2e9bd25806fcb9d5cf7740 WatchSource:0}: Error finding container 93aa7f92b9bcc08c666ee181695d56b209fb65b67c2e9bd25806fcb9d5cf7740: Status 404 returned error can't find the container with id 93aa7f92b9bcc08c666ee181695d56b209fb65b67c2e9bd25806fcb9d5cf7740 Nov 01 00:51:11.482897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:11.482860 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" event={"ID":"69ddff29-fa3e-49e0-b6f4-71c62187e64f","Type":"ContainerStarted","Data":"f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909"} Nov 01 00:51:11.484522 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:11.484492 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" event={"ID":"6d8804ef-0225-471a-8429-43442211dcdb","Type":"ContainerStarted","Data":"93aa7f92b9bcc08c666ee181695d56b209fb65b67c2e9bd25806fcb9d5cf7740"} Nov 01 00:51:11.605968 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:11.605914 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podStartSLOduration=2.605898856 podStartE2EDuration="2.605898856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:11.603038751 +0000 UTC m=+1661.582621077" watchObservedRunningTime="2026-11-01 00:51:11.605898856 +0000 UTC m=+1661.585481191" Nov 01 00:51:11.941258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:11.941163 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:11.947910 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:11.947885 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:12.401273 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:12.401228 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8804ef_0225_471a_8429_43442211dcdb.slice/crio-conmon-8f8aaa591ed7155e9511ec008ce5ae0a760f434b516d8d89c2c9fd55f0307eeb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8804ef_0225_471a_8429_43442211dcdb.slice/crio-8f8aaa591ed7155e9511ec008ce5ae0a760f434b516d8d89c2c9fd55f0307eeb.scope\": RecentStats: unable to find data in memory cache]" Nov 01 00:51:12.500428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:12.500396 4910 generic.go:356] "Generic (PLEG): container finished" podID="6d8804ef-0225-471a-8429-43442211dcdb" containerID="8f8aaa591ed7155e9511ec008ce5ae0a760f434b516d8d89c2c9fd55f0307eeb" exitCode=0 Nov 01 00:51:12.500638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:12.500480 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" event={"ID":"6d8804ef-0225-471a-8429-43442211dcdb","Type":"ContainerDied","Data":"8f8aaa591ed7155e9511ec008ce5ae0a760f434b516d8d89c2c9fd55f0307eeb"} Nov 01 00:51:12.500959 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:12.500939 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:12.502048 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:12.502028 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:51:13.106426 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:13.106383 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:51:13.682286 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.682257 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:13.767950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.767855 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8804ef-0225-471a-8429-43442211dcdb-kube-api-access-jgl42\" (UniqueName: \"kubernetes.io/projected/6d8804ef-0225-471a-8429-43442211dcdb-kube-api-access-jgl42\") pod \"6d8804ef-0225-471a-8429-43442211dcdb\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " Nov 01 00:51:13.767950 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.767931 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-openshift-ca\") pod \"6d8804ef-0225-471a-8429-43442211dcdb\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " Nov 01 00:51:13.768220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.768202 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-xks-ca\") pod \"6d8804ef-0225-471a-8429-43442211dcdb\" (UID: \"6d8804ef-0225-471a-8429-43442211dcdb\") " Nov 01 00:51:13.768295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.768252 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-openshift-ca" pod "6d8804ef-0225-471a-8429-43442211dcdb" (UID: "6d8804ef-0225-471a-8429-43442211dcdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:13.768760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.768740 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:13.770289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.770261 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8804ef-0225-471a-8429-43442211dcdb-kube-api-access-jgl42" pod "6d8804ef-0225-471a-8429-43442211dcdb" (UID: "6d8804ef-0225-471a-8429-43442211dcdb"). InnerVolumeSpecName "kube-api-access-jgl42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:13.770289 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.770284 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-xks-ca" pod "6d8804ef-0225-471a-8429-43442211dcdb" (UID: "6d8804ef-0225-471a-8429-43442211dcdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:51:13.870731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.870696 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d8804ef-0225-471a-8429-43442211dcdb-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:13.870731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:13.870735 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgl42\" (UniqueName: \"kubernetes.io/projected/6d8804ef-0225-471a-8429-43442211dcdb-kube-api-access-jgl42\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:14.520768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:14.520740 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" Nov 01 00:51:14.521271 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:14.520775 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2" event={"ID":"6d8804ef-0225-471a-8429-43442211dcdb","Type":"ContainerDied","Data":"93aa7f92b9bcc08c666ee181695d56b209fb65b67c2e9bd25806fcb9d5cf7740"} Nov 01 00:51:14.521271 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:14.520823 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93aa7f92b9bcc08c666ee181695d56b209fb65b67c2e9bd25806fcb9d5cf7740" Nov 01 00:51:18.276684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.276652 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2"] Nov 01 00:51:18.277330 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.277315 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6d8804ef-0225-471a-8429-43442211dcdb" containerName="revoke-keys" Nov 01 00:51:18.277373 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.277331 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8804ef-0225-471a-8429-43442211dcdb" containerName="revoke-keys" Nov 01 00:51:18.277443 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.277434 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8804ef-0225-471a-8429-43442211dcdb" containerName="revoke-keys" Nov 01 00:51:18.281459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.281443 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342328 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342298 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342489 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342365 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342489 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342460 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342595 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342575 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhzm\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-kube-api-access-lmhzm\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342734 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342719 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342771 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342764 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342809 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342789 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.342845 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.342809 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.395396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.395362 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2"] Nov 01 00:51:18.445131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445085 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445183 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445234 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445328 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhzm\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-kube-api-access-lmhzm\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445380 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445451 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445514 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445550 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.445658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.445576 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.446084 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.446056 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.446210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.446087 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-credential-socket\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.446363 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.446342 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-data\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.446599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.446567 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-certs\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.447293 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.447271 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istiod-ca-cert\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.448184 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.448161 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-envoy\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.448747 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.448724 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-podinfo\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.514702 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.514670 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhzm\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-kube-api-access-lmhzm\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.514854 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.514733 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-token\") pod \"e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:18.596814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:18.596716 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:19.037762 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:51:19.037725 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4e0ed5_9803_4d9e_9ac2_b3be268080c7.slice/crio-fbcda2736af8f21c2210daa416a2bb4a3f37abf16fd449bfffdbedd23f85b871 WatchSource:0}: Error finding container fbcda2736af8f21c2210daa416a2bb4a3f37abf16fd449bfffdbedd23f85b871: Status 404 returned error can't find the container with id fbcda2736af8f21c2210daa416a2bb4a3f37abf16fd449bfffdbedd23f85b871 Nov 01 00:51:19.039751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.039725 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2"] Nov 01 00:51:19.040503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.040471 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:19.040553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.040536 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:19.040585 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.040567 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:19.594112 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.594081 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" event={"ID":"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7","Type":"ContainerStarted","Data":"228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c"} Nov 01 00:51:19.594538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.594131 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" event={"ID":"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7","Type":"ContainerStarted","Data":"fbcda2736af8f21c2210daa416a2bb4a3f37abf16fd449bfffdbedd23f85b871"} Nov 01 00:51:19.597146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.597126 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:19.717420 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:19.717356 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podStartSLOduration=1.717338566 podStartE2EDuration="1.717338566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:19.713850888 +0000 UTC m=+1669.693433223" watchObservedRunningTime="2026-11-01 00:51:19.717338566 +0000 UTC m=+1669.696920903" Nov 01 00:51:20.598933 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:20.598898 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 01 00:51:20.599344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:20.598960 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 01 00:51:21.413383 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.413350 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r"] Nov 01 00:51:21.420279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.420242 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.482274 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.482240 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r"] Nov 01 00:51:21.521056 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.521027 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:51:21.521233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.521099 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:51:21.524890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.524866 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldthp\" (UniqueName: \"kubernetes.io/projected/a46e6670-e8ea-487b-b760-3a6d5ef34174-kube-api-access-ldthp\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.524982 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.524937 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.525232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.525210 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.597466 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.597425 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 01 00:51:21.597695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.597483 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 01 00:51:21.626660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.626608 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldthp\" (UniqueName: \"kubernetes.io/projected/a46e6670-e8ea-487b-b760-3a6d5ef34174-kube-api-access-ldthp\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.627055 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.626704 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.627055 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.626820 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.627326 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.627306 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.629232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.629205 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.693170 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.693096 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldthp\" (UniqueName: \"kubernetes.io/projected/a46e6670-e8ea-487b-b760-3a6d5ef34174-kube-api-access-ldthp\") pod \"maas-api-revoke-sub-5f08827e-nf95r\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:21.738694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:21.738648 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:22.185780 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:51:22.185746 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46e6670_e8ea_487b_b760_3a6d5ef34174.slice/crio-6c61bc3476304f408a3e3e2d0ef6edae9e6e11c0cbe375568467012f861b10a9 WatchSource:0}: Error finding container 6c61bc3476304f408a3e3e2d0ef6edae9e6e11c0cbe375568467012f861b10a9: Status 404 returned error can't find the container with id 6c61bc3476304f408a3e3e2d0ef6edae9e6e11c0cbe375568467012f861b10a9 Nov 01 00:51:22.200476 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.200440 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r"] Nov 01 00:51:22.388998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.388957 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs"] Nov 01 00:51:22.395045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.395016 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.492538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.492497 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs"] Nov 01 00:51:22.545259 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545221 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rhh4\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-kube-api-access-2rhh4\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.545436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545395 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.545512 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545490 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.545590 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545575 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.545801 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545779 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.545897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545882 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.545938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545916 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.545970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545944 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309de2b1-bd96-45e9-9e5c-5649dede132d-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.546007 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.545992 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.597669 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.597284 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 01 00:51:22.597669 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.597360 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 01 00:51:22.625715 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.625667 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" event={"ID":"a46e6670-e8ea-487b-b760-3a6d5ef34174","Type":"ContainerStarted","Data":"650f898e085aa868e61ea1e26b1ad1fdd1ffa8290eabc42cf739f1fdb6986926"} Nov 01 00:51:22.625905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.625742 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" event={"ID":"a46e6670-e8ea-487b-b760-3a6d5ef34174","Type":"ContainerStarted","Data":"6c61bc3476304f408a3e3e2d0ef6edae9e6e11c0cbe375568467012f861b10a9"} Nov 01 00:51:22.647246 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647216 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.647664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647293 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.647664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647355 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.647664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647382 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.647664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647404 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309de2b1-bd96-45e9-9e5c-5649dede132d-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.647664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647435 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.647664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647506 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rhh4\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-kube-api-access-2rhh4\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.648034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647665 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-credential-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.648034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647739 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-certs\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.648034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647750 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.648034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647770 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-data\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.648034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.647827 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.648443 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.648418 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-socket\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.648514 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.648456 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309de2b1-bd96-45e9-9e5c-5649dede132d-istiod-ca-cert\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.654238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.654148 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-podinfo\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.679416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.679058 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-envoy\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.736806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.719419 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-token\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.837188 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.835708 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rhh4\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-kube-api-access-2rhh4\") pod \"e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:22.852478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:22.852371 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" podStartSLOduration=1.8523535309999999 podStartE2EDuration="1.852353531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:22.836557684 +0000 UTC m=+1672.816140019" watchObservedRunningTime="2026-11-01 00:51:22.852353531 +0000 UTC m=+1672.831935901" Nov 01 00:51:23.009896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.009861 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:23.713658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.709272 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 01 00:51:23.713658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.709329 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 01 00:51:23.847736 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.847699 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs"] Nov 01 00:51:23.860865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.860181 4910 generic.go:356] "Generic (PLEG): container finished" podID="a46e6670-e8ea-487b-b760-3a6d5ef34174" containerID="650f898e085aa868e61ea1e26b1ad1fdd1ffa8290eabc42cf739f1fdb6986926" exitCode=0 Nov 01 00:51:23.860865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.860394 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" event={"ID":"a46e6670-e8ea-487b-b760-3a6d5ef34174","Type":"ContainerDied","Data":"650f898e085aa868e61ea1e26b1ad1fdd1ffa8290eabc42cf739f1fdb6986926"} Nov 01 00:51:23.895130 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.895059 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:23.895483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.895426 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:23.895769 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:23.895727 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:24.597820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:24.597781 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 01 00:51:24.597977 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:24.597864 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 01 00:51:24.911037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:24.910703 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" event={"ID":"309de2b1-bd96-45e9-9e5c-5649dede132d","Type":"ContainerStarted","Data":"94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b"} Nov 01 00:51:24.911037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:24.910801 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" event={"ID":"309de2b1-bd96-45e9-9e5c-5649dede132d","Type":"ContainerStarted","Data":"46b0a880ab42f7c04960faafdccb3025f19c23b23ca017356955696ea473c7d6"} Nov 01 00:51:25.012399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.011713 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:25.013332 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.013299 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 01 00:51:25.013440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.013370 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 01 00:51:25.044659 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.042609 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podStartSLOduration=3.042590067 podStartE2EDuration="3.042590067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:25.034919558 +0000 UTC m=+1675.014501908" watchObservedRunningTime="2026-11-01 00:51:25.042590067 +0000 UTC m=+1675.022172404" Nov 01 00:51:25.223548 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.223514 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:25.280415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.280378 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-xks-ca\") pod \"a46e6670-e8ea-487b-b760-3a6d5ef34174\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " Nov 01 00:51:25.280614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.280466 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46e6670-e8ea-487b-b760-3a6d5ef34174-kube-api-access-ldthp\" (UniqueName: \"kubernetes.io/projected/a46e6670-e8ea-487b-b760-3a6d5ef34174-kube-api-access-ldthp\") pod \"a46e6670-e8ea-487b-b760-3a6d5ef34174\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " Nov 01 00:51:25.280614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.280530 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-openshift-ca\") pod \"a46e6670-e8ea-487b-b760-3a6d5ef34174\" (UID: \"a46e6670-e8ea-487b-b760-3a6d5ef34174\") " Nov 01 00:51:25.281331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.281304 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-openshift-ca" pod "a46e6670-e8ea-487b-b760-3a6d5ef34174" (UID: "a46e6670-e8ea-487b-b760-3a6d5ef34174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:25.282850 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.282820 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46e6670-e8ea-487b-b760-3a6d5ef34174-kube-api-access-ldthp" pod "a46e6670-e8ea-487b-b760-3a6d5ef34174" (UID: "a46e6670-e8ea-487b-b760-3a6d5ef34174"). InnerVolumeSpecName "kube-api-access-ldthp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:25.282850 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.282821 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-xks-ca" pod "a46e6670-e8ea-487b-b760-3a6d5ef34174" (UID: "a46e6670-e8ea-487b-b760-3a6d5ef34174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:51:25.390617 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.390577 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:25.400661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.390941 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldthp\" (UniqueName: \"kubernetes.io/projected/a46e6670-e8ea-487b-b760-3a6d5ef34174-kube-api-access-ldthp\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:25.400902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.400872 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46e6670-e8ea-487b-b760-3a6d5ef34174-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:25.597516 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.597478 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 01 00:51:25.597708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.597567 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 01 00:51:25.925164 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.925069 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" event={"ID":"a46e6670-e8ea-487b-b760-3a6d5ef34174","Type":"ContainerDied","Data":"6c61bc3476304f408a3e3e2d0ef6edae9e6e11c0cbe375568467012f861b10a9"} Nov 01 00:51:25.925164 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.925097 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r" Nov 01 00:51:25.925164 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:25.925125 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c61bc3476304f408a3e3e2d0ef6edae9e6e11c0cbe375568467012f861b10a9" Nov 01 00:51:26.012757 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.012722 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 01 00:51:26.012969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.012783 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 01 00:51:26.696872 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.696836 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr"] Nov 01 00:51:26.697202 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.697169 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" containerID="cri-o://481cbd4226e508de0b7df0109be53d7853fad31101c25efa6b6f23afdbb6c7ac" gracePeriod=30 Nov 01 00:51:26.745947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.745903 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6"] Nov 01 00:51:26.746174 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.746146 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" containerID="cri-o://6abcb4bc28096cac36e2b2d7e1e99f6bbeaac66752ea034322a99764f960f524" gracePeriod=30 Nov 01 00:51:26.875650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.861084 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:26.892932 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.892904 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:26.899665 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:26.899589 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 00:51:27.010184 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:27.010153 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 01 00:51:27.010601 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:27.010200 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 01 00:51:28.010619 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:28.010581 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 01 00:51:28.011097 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:28.010662 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 01 00:51:29.010836 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:29.010798 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 01 00:51:29.011257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:29.010876 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 01 00:51:30.010748 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.010713 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 01 00:51:30.010927 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.010787 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 01 00:51:30.524733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.524687 4910 scope.go:122] "RemoveContainer" containerID="41f2b45ec37a8153a18a60a3eb0b12a21709bea8d5cf567df7d072c948c642a3" Nov 01 00:51:30.544336 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.544313 4910 scope.go:122] "RemoveContainer" containerID="041dd3ca133e0d80358322334ade4d907749cd0dc5e14366b68174e45c62b398" Nov 01 00:51:30.555837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.555814 4910 scope.go:122] "RemoveContainer" containerID="a7a0763f72c124d166f1881a36816efef984829a38b20ce3c38f7cd944309ea1" Nov 01 00:51:30.568690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.568667 4910 scope.go:122] "RemoveContainer" containerID="36d3ee2c45a29595e1fa2b11292b14fa12b197a85a7a7a85c244ec7e98ad6e31" Nov 01 00:51:30.580907 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.580887 4910 scope.go:122] "RemoveContainer" containerID="b3aaee0379bdde739a58cb278223e08b2c2e9293515fc1e0e703abe0a32d9d0f" Nov 01 00:51:30.607891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.607866 4910 scope.go:122] "RemoveContainer" containerID="573eaef2a18661115b7b55d9fb7c025c2f0b8fea9d48000fecdfe25c904fa154" Nov 01 00:51:30.622855 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.622827 4910 scope.go:122] "RemoveContainer" containerID="abadbca92e896c69c22557e8acfc37955653c71172373e3562d240bba0f063cd" Nov 01 00:51:30.635299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.635273 4910 scope.go:122] "RemoveContainer" containerID="84ce11e89e9459255584767bb614052aa8bf54c260ec8a2ddbeb9d8e46d285cb" Nov 01 00:51:30.646599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:30.646579 4910 scope.go:122] "RemoveContainer" containerID="8b50ac040979cd653e8c615a86ad8b0f4510f794adecce7e378b70e4d330080b" Nov 01 00:51:31.010869 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.010833 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 01 00:51:31.011044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.010898 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 01 00:51:31.365263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.365155 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz"] Nov 01 00:51:31.366343 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.366312 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a46e6670-e8ea-487b-b760-3a6d5ef34174" containerName="revoke-keys" Nov 01 00:51:31.366343 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.366342 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46e6670-e8ea-487b-b760-3a6d5ef34174" containerName="revoke-keys" Nov 01 00:51:31.366559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.366539 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46e6670-e8ea-487b-b760-3a6d5ef34174" containerName="revoke-keys" Nov 01 00:51:31.373568 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.373427 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.428846 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.428811 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz"] Nov 01 00:51:31.469898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.469770 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:51:31.469898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.469840 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:51:31.532129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.532088 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.532129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.532130 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.532411 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.532276 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjlh\" (UniqueName: \"kubernetes.io/projected/63d3cff4-6863-454c-ad90-7e9a93d291b9-kube-api-access-jkjlh\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.634311 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.634212 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.634311 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.634268 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.634556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.634377 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjlh\" (UniqueName: \"kubernetes.io/projected/63d3cff4-6863-454c-ad90-7e9a93d291b9-kube-api-access-jkjlh\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.635063 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.635038 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.636886 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.636865 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.696272 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.696240 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjlh\" (UniqueName: \"kubernetes.io/projected/63d3cff4-6863-454c-ad90-7e9a93d291b9-kube-api-access-jkjlh\") pod \"maas-api-revoke-sub-01b3da1b-w7wnz\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:31.696523 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:31.696506 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:32.015376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.015339 4910 generic.go:356] "Generic (PLEG): container finished" podID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerID="481cbd4226e508de0b7df0109be53d7853fad31101c25efa6b6f23afdbb6c7ac" exitCode=0 Nov 01 00:51:32.015750 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.015712 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" event={"ID":"c2f7c43a-37d1-446f-b8c6-445a71575fe8","Type":"ContainerDied","Data":"481cbd4226e508de0b7df0109be53d7853fad31101c25efa6b6f23afdbb6c7ac"} Nov 01 00:51:32.016903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.016881 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:32.017700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.017679 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:32.018798 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.018774 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:51:32.020031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.020001 4910 generic.go:356] "Generic (PLEG): container finished" podID="c6096227-de28-416a-a1cf-909227cd52d9" containerID="6abcb4bc28096cac36e2b2d7e1e99f6bbeaac66752ea034322a99764f960f524" exitCode=0 Nov 01 00:51:32.020141 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.020125 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" event={"ID":"c6096227-de28-416a-a1cf-909227cd52d9","Type":"ContainerDied","Data":"6abcb4bc28096cac36e2b2d7e1e99f6bbeaac66752ea034322a99764f960f524"} Nov 01 00:51:32.023423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.023403 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:51:32.050754 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.050724 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:51:32.118860 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:51:32.118822 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d3cff4_6863_454c_ad90_7e9a93d291b9.slice/crio-69719f6879701edd7b9fca99824da40fd4da46608b69e076d8b5fba864b5177c WatchSource:0}: Error finding container 69719f6879701edd7b9fca99824da40fd4da46608b69e076d8b5fba864b5177c: Status 404 returned error can't find the container with id 69719f6879701edd7b9fca99824da40fd4da46608b69e076d8b5fba864b5177c Nov 01 00:51:32.119991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.119964 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz"] Nov 01 00:51:32.142547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.142517 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-credential-socket\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.142720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.142560 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istiod-ca-cert\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.142720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.142590 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-kube-api-access-f7pjw\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-kube-api-access-f7pjw\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.142720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.142614 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6096227-de28-416a-a1cf-909227cd52d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6096227-de28-416a-a1cf-909227cd52d9-istiod-ca-cert\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.142720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.142709 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-data\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.142905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.142752 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-podinfo\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.142905 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.142838 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-credential-socket" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.143061 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143038 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-data" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.143135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143105 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6096227-de28-416a-a1cf-909227cd52d9-istiod-ca-cert" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:32.143355 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143322 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istiod-ca-cert" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:32.143355 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143323 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-istio-token\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-istio-token\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.143499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143440 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-certs\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.143499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143486 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-envoy\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.143588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143558 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-token\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-token\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.143820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.143802 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-certs\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.144093 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144069 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-socket\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.144156 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144117 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-data\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.144205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144190 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-socket\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.144250 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144236 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-credential-socket\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.144287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144277 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6096227-de28-416a-a1cf-909227cd52d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6096227-de28-416a-a1cf-909227cd52d9-istio-podinfo\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.144353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144337 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-kube-api-access-kgzz5\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-kube-api-access-kgzz5\") pod \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\" (UID: \"c2f7c43a-37d1-446f-b8c6-445a71575fe8\") " Nov 01 00:51:32.144432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144418 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-envoy\") pod \"c6096227-de28-416a-a1cf-909227cd52d9\" (UID: \"c6096227-de28-416a-a1cf-909227cd52d9\") " Nov 01 00:51:32.144917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.144896 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-credential-socket" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.145299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145264 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-istio-token" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:32.145525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145501 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.145525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145522 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.145720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145532 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.145720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145540 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6096227-de28-416a-a1cf-909227cd52d9-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.145720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145549 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.145720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145557 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.145720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145503 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-kube-api-access-f7pjw" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "kube-api-access-f7pjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:32.145720 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145566 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-podinfo" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:51:32.145970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145801 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-socket" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.145970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.145879 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-socket" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.146033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.146006 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-certs" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.146224 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.146199 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-data" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.146224 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.146211 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-certs" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.148563 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.148527 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-kube-api-access-kgzz5" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "kube-api-access-kgzz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:32.148563 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.148528 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6096227-de28-416a-a1cf-909227cd52d9-istio-podinfo" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:51:32.148563 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.148558 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-token" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:32.148807 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.148607 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-envoy" pod "c6096227-de28-416a-a1cf-909227cd52d9" (UID: "c6096227-de28-416a-a1cf-909227cd52d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.150098 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.150077 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-envoy" pod "c2f7c43a-37d1-446f-b8c6-445a71575fe8" (UID: "c2f7c43a-37d1-446f-b8c6-445a71575fe8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:51:32.246705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246675 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246702 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246713 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246722 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6096227-de28-416a-a1cf-909227cd52d9-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246731 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzz5\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-kube-api-access-kgzz5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246742 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246750 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7pjw\" (UniqueName: \"kubernetes.io/projected/c6096227-de28-416a-a1cf-909227cd52d9-kube-api-access-f7pjw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246758 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246766 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246774 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246784 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f7c43a-37d1-446f-b8c6-445a71575fe8-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:32.246953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:32.246793 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6096227-de28-416a-a1cf-909227cd52d9-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:33.030163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.030134 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" Nov 01 00:51:33.030163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.030155 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr" event={"ID":"c2f7c43a-37d1-446f-b8c6-445a71575fe8","Type":"ContainerDied","Data":"f7c197dddf02a7eb657718ccf3167b0424116b9da0d78866a0902f659194c472"} Nov 01 00:51:33.030717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.030221 4910 scope.go:122] "RemoveContainer" containerID="481cbd4226e508de0b7df0109be53d7853fad31101c25efa6b6f23afdbb6c7ac" Nov 01 00:51:33.032971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.032276 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" event={"ID":"63d3cff4-6863-454c-ad90-7e9a93d291b9","Type":"ContainerStarted","Data":"53a6902290c965978be00f598ff00feae78c18d8541cdb636637ef8a1511fdc9"} Nov 01 00:51:33.032971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.032816 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" event={"ID":"63d3cff4-6863-454c-ad90-7e9a93d291b9","Type":"ContainerStarted","Data":"69719f6879701edd7b9fca99824da40fd4da46608b69e076d8b5fba864b5177c"} Nov 01 00:51:33.034072 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.034056 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" Nov 01 00:51:33.034173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.034076 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6" event={"ID":"c6096227-de28-416a-a1cf-909227cd52d9","Type":"ContainerDied","Data":"d1d66d33ed1cfe2a3de56b728ccf49e952598531f08508ffb69f680f9441a76c"} Nov 01 00:51:33.043026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.043007 4910 scope.go:122] "RemoveContainer" containerID="6abcb4bc28096cac36e2b2d7e1e99f6bbeaac66752ea034322a99764f960f524" Nov 01 00:51:33.204521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.204466 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" podStartSLOduration=2.204450076 podStartE2EDuration="2.204450076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:33.199256478 +0000 UTC m=+1683.178838813" watchObservedRunningTime="2026-11-01 00:51:33.204450076 +0000 UTC m=+1683.184032410" Nov 01 00:51:33.317231 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.317142 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6"] Nov 01 00:51:33.372690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.370228 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cn8t6"] Nov 01 00:51:33.478170 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.478134 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr"] Nov 01 00:51:33.483978 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:33.483944 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-fvzwr"] Nov 01 00:51:34.057247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:34.057211 4910 generic.go:356] "Generic (PLEG): container finished" podID="63d3cff4-6863-454c-ad90-7e9a93d291b9" containerID="53a6902290c965978be00f598ff00feae78c18d8541cdb636637ef8a1511fdc9" exitCode=0 Nov 01 00:51:34.057828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:34.057301 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" event={"ID":"63d3cff4-6863-454c-ad90-7e9a93d291b9","Type":"ContainerDied","Data":"53a6902290c965978be00f598ff00feae78c18d8541cdb636637ef8a1511fdc9"} Nov 01 00:51:34.197704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:34.197669 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" path="/var/lib/kubelet/pods/c2f7c43a-37d1-446f-b8c6-445a71575fe8/volumes" Nov 01 00:51:34.198220 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:34.198203 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6096227-de28-416a-a1cf-909227cd52d9" path="/var/lib/kubelet/pods/c6096227-de28-416a-a1cf-909227cd52d9/volumes" Nov 01 00:51:35.258274 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.258234 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:35.298339 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.298284 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d3cff4-6863-454c-ad90-7e9a93d291b9-kube-api-access-jkjlh\" (UniqueName: \"kubernetes.io/projected/63d3cff4-6863-454c-ad90-7e9a93d291b9-kube-api-access-jkjlh\") pod \"63d3cff4-6863-454c-ad90-7e9a93d291b9\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " Nov 01 00:51:35.298553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.298428 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-xks-ca\") pod \"63d3cff4-6863-454c-ad90-7e9a93d291b9\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " Nov 01 00:51:35.298553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.298504 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-openshift-ca\") pod \"63d3cff4-6863-454c-ad90-7e9a93d291b9\" (UID: \"63d3cff4-6863-454c-ad90-7e9a93d291b9\") " Nov 01 00:51:35.298870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.298843 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-openshift-ca" pod "63d3cff4-6863-454c-ad90-7e9a93d291b9" (UID: "63d3cff4-6863-454c-ad90-7e9a93d291b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:51:35.299879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.299853 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:35.300876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.300847 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d3cff4-6863-454c-ad90-7e9a93d291b9-kube-api-access-jkjlh" pod "63d3cff4-6863-454c-ad90-7e9a93d291b9" (UID: "63d3cff4-6863-454c-ad90-7e9a93d291b9"). InnerVolumeSpecName "kube-api-access-jkjlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:51:35.300876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.300868 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-xks-ca" pod "63d3cff4-6863-454c-ad90-7e9a93d291b9" (UID: "63d3cff4-6863-454c-ad90-7e9a93d291b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:51:35.401976 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.401882 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjlh\" (UniqueName: \"kubernetes.io/projected/63d3cff4-6863-454c-ad90-7e9a93d291b9-kube-api-access-jkjlh\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:35.401976 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.401924 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d3cff4-6863-454c-ad90-7e9a93d291b9-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:51:35.602056 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.602017 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78"] Nov 01 00:51:35.603044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603021 4910 cpu_manager.go:414] "containerMap: removing container" podUID="63d3cff4-6863-454c-ad90-7e9a93d291b9" containerName="revoke-keys" Nov 01 00:51:35.603120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603046 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d3cff4-6863-454c-ad90-7e9a93d291b9" containerName="revoke-keys" Nov 01 00:51:35.603120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603065 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" Nov 01 00:51:35.603120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603073 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" Nov 01 00:51:35.603295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603177 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" Nov 01 00:51:35.603295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603185 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" Nov 01 00:51:35.603396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603332 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f7c43a-37d1-446f-b8c6-445a71575fe8" containerName="istio-proxy" Nov 01 00:51:35.603396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603344 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d3cff4-6863-454c-ad90-7e9a93d291b9" containerName="revoke-keys" Nov 01 00:51:35.603396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.603353 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6096227-de28-416a-a1cf-909227cd52d9" containerName="istio-proxy" Nov 01 00:51:35.607941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.607917 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.671832 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.671749 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78"] Nov 01 00:51:35.706810 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.706753 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-istio-token\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.706821 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-data\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.706897 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-credential-socket\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.706990 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15ef169-84ef-4678-b219-641efe029c30-istio-podinfo\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.707024 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-socket\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.707050 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbg5z\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-kube-api-access-dbg5z\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.707106 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15ef169-84ef-4678-b219-641efe029c30-istiod-ca-cert\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707338 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.707220 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-certs\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.707338 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.707316 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-envoy\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.750239 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.750209 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-14957a74-gw-openshift-default-dockercfg-zrn8m\"" Nov 01 00:51:35.861655 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.845927 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-credential-socket\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.862057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.862031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15ef169-84ef-4678-b219-641efe029c30-istio-podinfo\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.872715 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-socket\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.872761 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbg5z\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-kube-api-access-dbg5z\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.872825 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15ef169-84ef-4678-b219-641efe029c30-istiod-ca-cert\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.872961 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-certs\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.873070 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-envoy\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.873137 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-istio-token\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.873187 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-data\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.873603 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-data\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.858051 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-credential-socket\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.873902 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-socket\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.874833 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15ef169-84ef-4678-b219-641efe029c30-istiod-ca-cert\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.898966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.875063 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-certs\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.922244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.922170 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-envoy\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.926849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.924278 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15ef169-84ef-4678-b219-641efe029c30-istio-podinfo\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.949160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.949090 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbg5z\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-kube-api-access-dbg5z\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:35.953194 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:35.953132 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-istio-token\") pod \"e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:36.082585 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.082544 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" event={"ID":"63d3cff4-6863-454c-ad90-7e9a93d291b9","Type":"ContainerDied","Data":"69719f6879701edd7b9fca99824da40fd4da46608b69e076d8b5fba864b5177c"} Nov 01 00:51:36.082806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.082605 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69719f6879701edd7b9fca99824da40fd4da46608b69e076d8b5fba864b5177c" Nov 01 00:51:36.082806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.082694 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz" Nov 01 00:51:36.225652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.225561 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:36.674916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.673261 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78"] Nov 01 00:51:36.711735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.697800 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:36.711735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.697913 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:36.711735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:36.697965 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:37.415648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:37.393991 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" event={"ID":"c15ef169-84ef-4678-b219-641efe029c30","Type":"ContainerStarted","Data":"ba74432c7e52ec425f48319ca19f8b0191c139e92d54b7adc5c107486af8c727"} Nov 01 00:51:37.415648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:37.394071 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" event={"ID":"c15ef169-84ef-4678-b219-641efe029c30","Type":"ContainerStarted","Data":"747f999719908f81ba6a4ff4fb3d5973023913774eaa39cf93713a919bfba10c"} Nov 01 00:51:38.228061 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:38.228030 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:38.228788 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:38.228764 4910 patch_prober.go:29] interesting pod/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 00:51:38.228983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:38.228960 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 00:51:39.226117 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:39.226073 4910 patch_prober.go:29] interesting pod/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 00:51:39.226423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:39.226390 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 00:51:39.656899 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:39.656858 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:51:39.727653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:39.715995 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.727653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:39.716402 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:39.727653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:39.717583 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:39.848225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:39.848173 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.617656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.600609 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:40.619088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.619046 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:51:40.693806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.693753 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.715667 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.707133 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:51:40.773647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.729784 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.773647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.733216 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.773647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.742357 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:40.773647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.754747 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:51:40.773647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.754801 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:51:40.773647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.746883 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:40.773647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.755650 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:51:40.970708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:40.947508 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podStartSLOduration=5.94748341 podStartE2EDuration="5.94748341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:37.544298761 +0000 UTC m=+1687.523881100" watchObservedRunningTime="2026-11-01 00:51:40.94748341 +0000 UTC m=+1690.927065746" Nov 01 00:51:41.077650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.075132 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:51:41.077650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.077344 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:51:41.230709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.216280 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:41.230709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.225254 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:51:41.387818 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.387773 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.419389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.419345 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:41.419708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.419676 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.461205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.461163 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:41.461493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.461463 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.635856 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.635811 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:51:41.637901 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.637874 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:51:41.734650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.711424 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:51:41.805346 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.755187 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:41.805877 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.755224 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:41.806503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:41.755254 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.152683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:42.078053 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.440711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:42.416976 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 00:51:42.714088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:42.693362 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.875829 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:42.875779 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.923844 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:42.923790 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:42.963127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:42.963085 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.045648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.039598 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:43.045648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.039674 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.082650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.074125 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:51:43.262652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.259449 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.360742 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.351711 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 01 00:51:43.447650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.441299 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.466648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.450340 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:51:43.722308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.722197 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:43.848647 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:51:43.827808 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:51:43.941651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.937754 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.006257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.955436 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 01 00:51:44.007228 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:43.976029 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 00:51:44.183085 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.183037 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.265356 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.265267 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:44.265616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.265587 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.291848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.291798 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:44.292162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.292114 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.311814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.311776 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:51:44.312123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.312089 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.415653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.411704 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.457652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.456957 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:44.732150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.732044 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:51:44.778649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.778180 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Nov 01 00:51:44.782691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.780431 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:51:44.817674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:44.812905 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:51:45.016252 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:45.015461 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:51:45.021858 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:45.021824 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:51:45.022435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:45.022409 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 01 00:51:45.049390 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:45.049134 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:51:45.051403 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:45.050553 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:51:45.051403 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:45.050602 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:51:45.052459 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:45.052439 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 00:51:48.516103 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.516066 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7"] Nov 01 00:51:48.521976 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.521950 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.575306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.575275 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 00:51:48.575511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.575389 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 00:51:48.575511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.575431 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 00:51:48.575511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.575445 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lqx6r\"" Nov 01 00:51:48.604690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.589343 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7"] Nov 01 00:51:48.611716 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.611685 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.611891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.611730 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.611891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.611816 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.611891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.611881 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdb4\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-kube-api-access-vxdb4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.612113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.611910 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.612113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.611936 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.612113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.612105 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.713884 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.713850 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.714068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.714037 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.714196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.714180 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.714244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.714228 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.714310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.714296 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.714365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.714337 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdb4\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-kube-api-access-vxdb4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.714365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.714356 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.715138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.715099 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.716794 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.716764 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.716900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.716813 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.717181 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.717161 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.717243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.717165 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.777394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.777304 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.777561 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.777393 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdb4\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-kube-api-access-vxdb4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzkr7\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:48.837037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:48.837001 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:49.499311 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:49.499273 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7"] Nov 01 00:51:49.530154 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:51:49.530120 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3a555_9cc3_49d3_b62d_db6c0b3519e5.slice/crio-1df3c854b9b750f185a3558edea10d9e17642a1a9ff7247793a1a472809e881a WatchSource:0}: Error finding container 1df3c854b9b750f185a3558edea10d9e17642a1a9ff7247793a1a472809e881a: Status 404 returned error can't find the container with id 1df3c854b9b750f185a3558edea10d9e17642a1a9ff7247793a1a472809e881a Nov 01 00:51:49.623120 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:49.622984 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" event={"ID":"50f3a555-9cc3-49d3-b62d-db6c0b3519e5","Type":"ContainerStarted","Data":"1df3c854b9b750f185a3558edea10d9e17642a1a9ff7247793a1a472809e881a"} Nov 01 00:51:53.607184 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:53.607152 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:53.607486 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:53.607216 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:54.691392 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:54.691352 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" event={"ID":"50f3a555-9cc3-49d3-b62d-db6c0b3519e5","Type":"ContainerStarted","Data":"8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e"} Nov 01 00:51:54.916607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:54.916537 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" podStartSLOduration=2.842946967 podStartE2EDuration="6.916520476s" totalImagesPullingTime="4.073573509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:54.865382328 +0000 UTC m=+1704.844964696" watchObservedRunningTime="2026-11-01 00:51:54.916520476 +0000 UTC m=+1704.896102811" Nov 01 00:51:55.703636 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:55.703590 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:55.705704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:55.705680 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 00:51:57.048320 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.048288 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75"] Nov 01 00:51:57.053139 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.053116 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.165854 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.165805 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166028 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.165860 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5fm\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-kube-api-access-9p5fm\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166028 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.165975 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166141 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.166122 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.166161 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.166237 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.166414 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee64569-1fa7-4c78-b25f-32cefd90e832-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.166472 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.166521 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.166501 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.267930 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.267892 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.268217 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268186 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee64569-1fa7-4c78-b25f-32cefd90e832-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.268380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268263 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.268380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268297 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.268380 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268378 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.268569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268407 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5fm\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-kube-api-access-9p5fm\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.268569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268448 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.268569 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268465 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.270551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.268974 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.270551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.269022 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.270551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.269068 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.270551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.270229 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee64569-1fa7-4c78-b25f-32cefd90e832-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.270900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.270878 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75"] Nov 01 00:51:57.271233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.271214 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.271348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.271243 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.271448 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.271373 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.273170 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.273148 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.330368 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.330294 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5fm\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-kube-api-access-9p5fm\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.331717 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.331695 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-d6f75\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.368150 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.368112 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:57.821418 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.821376 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75"] Nov 01 00:51:57.824011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.823973 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:57.824108 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.824071 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:57.824151 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:57.824116 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:51:58.731880 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:58.731844 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" event={"ID":"dee64569-1fa7-4c78-b25f-32cefd90e832","Type":"ContainerStarted","Data":"876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71"} Nov 01 00:51:58.732414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:58.731907 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" event={"ID":"dee64569-1fa7-4c78-b25f-32cefd90e832","Type":"ContainerStarted","Data":"ab7d8f518e516a8535e6334659a64d1588601946b3fd61c4b8688ca81b20fb2f"} Nov 01 00:51:58.907925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:58.907873 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podStartSLOduration=2.907859211 podStartE2EDuration="2.907859211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:51:58.902854254 +0000 UTC m=+1708.882436587" watchObservedRunningTime="2026-11-01 00:51:58.907859211 +0000 UTC m=+1708.887441546" Nov 01 00:51:59.374799 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:59.374765 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:51:59.377019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:59.376993 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-d6f75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 00:51:59.377113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:51:59.377047 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 00:52:00.372487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.371740 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-d6f75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 00:52:00.372487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.371836 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 00:52:00.925470 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.925426 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg"] Nov 01 00:52:00.930408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.930385 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:00.943351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.943325 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flks\" (UniqueName: \"kubernetes.io/projected/fa9d9984-6e2e-4316-910e-82da61e64201-kube-api-access-6flks\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:00.943575 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.943555 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:00.943758 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.943738 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:00.984706 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:00.983864 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:52:01.036110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.036083 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:52:01.044280 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.044250 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg"] Nov 01 00:52:01.046752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.046724 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:01.046888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.046870 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flks\" (UniqueName: \"kubernetes.io/projected/fa9d9984-6e2e-4316-910e-82da61e64201-kube-api-access-6flks\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:01.047018 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.047001 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:01.048033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.048009 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:01.080872 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.073695 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:01.108941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.108895 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flks\" (UniqueName: \"kubernetes.io/projected/fa9d9984-6e2e-4316-910e-82da61e64201-kube-api-access-6flks\") pod \"maas-api-revoke-sub-28af79cc-k6rgg\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:01.248125 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.247861 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:01.377650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.374134 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-d6f75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 00:52:01.377650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.374194 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 00:52:01.713789 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.713756 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg"] Nov 01 00:52:01.796066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:01.796031 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" event={"ID":"fa9d9984-6e2e-4316-910e-82da61e64201","Type":"ContainerStarted","Data":"f46cb1fd6e52e820f8fa4c1ac50b66eae1c9357c31d1af3f2ed8910cb6fd56f0"} Nov 01 00:52:02.372648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:02.372023 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-d6f75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 00:52:02.372648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:02.372083 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 00:52:02.813586 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:02.812495 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" event={"ID":"fa9d9984-6e2e-4316-910e-82da61e64201","Type":"ContainerStarted","Data":"405f2d6115f25571c3b38d1e3cfac89e906de2fa2956b0018a32d603126f2d89"} Nov 01 00:52:02.939298 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:02.939235 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" podStartSLOduration=2.9392173010000002 podStartE2EDuration="2.939217301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:02.933131919 +0000 UTC m=+1712.912714254" watchObservedRunningTime="2026-11-01 00:52:02.939217301 +0000 UTC m=+1712.918799642" Nov 01 00:52:03.372763 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:03.372724 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-d6f75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 00:52:03.373041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:03.373009 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 00:52:03.846136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:03.845971 4910 generic.go:356] "Generic (PLEG): container finished" podID="fa9d9984-6e2e-4316-910e-82da61e64201" containerID="405f2d6115f25571c3b38d1e3cfac89e906de2fa2956b0018a32d603126f2d89" exitCode=0 Nov 01 00:52:03.846559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:03.846346 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" event={"ID":"fa9d9984-6e2e-4316-910e-82da61e64201","Type":"ContainerDied","Data":"405f2d6115f25571c3b38d1e3cfac89e906de2fa2956b0018a32d603126f2d89"} Nov 01 00:52:04.369068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:04.369028 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-d6f75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 00:52:04.369259 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:04.369114 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 00:52:05.129129 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.129049 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:05.278033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.277996 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-xks-ca\") pod \"fa9d9984-6e2e-4316-910e-82da61e64201\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " Nov 01 00:52:05.278208 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.278053 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-openshift-ca\") pod \"fa9d9984-6e2e-4316-910e-82da61e64201\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " Nov 01 00:52:05.278403 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.278377 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9d9984-6e2e-4316-910e-82da61e64201-kube-api-access-6flks\" (UniqueName: \"kubernetes.io/projected/fa9d9984-6e2e-4316-910e-82da61e64201-kube-api-access-6flks\") pod \"fa9d9984-6e2e-4316-910e-82da61e64201\" (UID: \"fa9d9984-6e2e-4316-910e-82da61e64201\") " Nov 01 00:52:05.278485 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.278403 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-openshift-ca" pod "fa9d9984-6e2e-4316-910e-82da61e64201" (UID: "fa9d9984-6e2e-4316-910e-82da61e64201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:52:05.279132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.279109 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:05.281327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.281293 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-xks-ca" pod "fa9d9984-6e2e-4316-910e-82da61e64201" (UID: "fa9d9984-6e2e-4316-910e-82da61e64201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:52:05.281327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.281303 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9d9984-6e2e-4316-910e-82da61e64201-kube-api-access-6flks" pod "fa9d9984-6e2e-4316-910e-82da61e64201" (UID: "fa9d9984-6e2e-4316-910e-82da61e64201"). InnerVolumeSpecName "kube-api-access-6flks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:52:05.369337 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.369298 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-d6f75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 00:52:05.369575 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.369357 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 00:52:05.380487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.380405 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9d9984-6e2e-4316-910e-82da61e64201-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:05.380487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.380438 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6flks\" (UniqueName: \"kubernetes.io/projected/fa9d9984-6e2e-4316-910e-82da61e64201-kube-api-access-6flks\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:05.871473 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.871436 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" event={"ID":"fa9d9984-6e2e-4316-910e-82da61e64201","Type":"ContainerDied","Data":"f46cb1fd6e52e820f8fa4c1ac50b66eae1c9357c31d1af3f2ed8910cb6fd56f0"} Nov 01 00:52:05.871473 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.871462 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg" Nov 01 00:52:05.871736 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:05.871528 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46cb1fd6e52e820f8fa4c1ac50b66eae1c9357c31d1af3f2ed8910cb6fd56f0" Nov 01 00:52:06.587135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:06.587106 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:52:06.587658 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:06.587615 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:52:06.588350 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:06.588330 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 00:52:11.679895 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.679858 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5"] Nov 01 00:52:11.680577 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.680561 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fa9d9984-6e2e-4316-910e-82da61e64201" containerName="revoke-keys" Nov 01 00:52:11.680691 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.680577 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa9d9984-6e2e-4316-910e-82da61e64201" containerName="revoke-keys" Nov 01 00:52:11.680755 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.680730 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa9d9984-6e2e-4316-910e-82da61e64201" containerName="revoke-keys" Nov 01 00:52:11.685681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.685660 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.738409 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.738379 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:52:11.738582 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.738521 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:52:11.742349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.742322 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5"] Nov 01 00:52:11.879454 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.879414 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15d641-1837-462a-8743-661c7871f95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.879701 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.879681 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkw8\" (UniqueName: \"kubernetes.io/projected/1b15d641-1837-462a-8743-661c7871f95b-kube-api-access-ktkw8\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.879904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.879879 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15d641-1837-462a-8743-661c7871f95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.981118 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.981029 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkw8\" (UniqueName: \"kubernetes.io/projected/1b15d641-1837-462a-8743-661c7871f95b-kube-api-access-ktkw8\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.981277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.981132 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15d641-1837-462a-8743-661c7871f95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.981277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.981209 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15d641-1837-462a-8743-661c7871f95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.981953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.981855 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15d641-1837-462a-8743-661c7871f95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:11.983741 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:11.983715 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15d641-1837-462a-8743-661c7871f95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:12.042558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.042532 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkw8\" (UniqueName: \"kubernetes.io/projected/1b15d641-1837-462a-8743-661c7871f95b-kube-api-access-ktkw8\") pod \"maas-api-revoke-sub-31ec1452-tpjh5\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:12.172883 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.172849 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5"] Nov 01 00:52:12.178925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.178905 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286098 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286069 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286118 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqt6l\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-kube-api-access-fqt6l\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286191 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286215 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286235 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286448 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286422 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286590 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286564 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286608 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.286697 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.286688 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.302167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.302137 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.388780 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.388839 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqt6l\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-kube-api-access-fqt6l\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.388873 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.388896 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.388923 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.388991 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.389068 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.389105 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.389164 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.390107 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.390376 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.391548 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.391843 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.393041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.392537 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.411558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.407529 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5"] Nov 01 00:52:12.411558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.410820 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.411558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.411452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.452821 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.452763 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.453781 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.453736 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqt6l\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-kube-api-access-fqt6l\") pod \"maas-default-gateway-openshift-default-67bf986f99-cfbd5\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.502100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.500849 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:12.706104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.706067 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5"] Nov 01 00:52:12.707058 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:52:12.707034 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b15d641_1837_462a_8743_661c7871f95b.slice/crio-a28e626707f3fb86f3535b54ae661f69761df58381755b0f182283dde12ac813 WatchSource:0}: Error finding container a28e626707f3fb86f3535b54ae661f69761df58381755b0f182283dde12ac813: Status 404 returned error can't find the container with id a28e626707f3fb86f3535b54ae661f69761df58381755b0f182283dde12ac813 Nov 01 00:52:12.893163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.893135 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5"] Nov 01 00:52:12.893381 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:52:12.893357 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2dd317_384d_4cbe_b4f3_2cd8cf04858a.slice/crio-8a015a261e003cc010d36ee86fc3840c6229cae8d51c7fcdf58b93ff7d153e51 WatchSource:0}: Error finding container 8a015a261e003cc010d36ee86fc3840c6229cae8d51c7fcdf58b93ff7d153e51: Status 404 returned error can't find the container with id 8a015a261e003cc010d36ee86fc3840c6229cae8d51c7fcdf58b93ff7d153e51 Nov 01 00:52:12.896807 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.896760 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:12.896936 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.896854 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:12.896936 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.896893 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:12.978011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.977964 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" event={"ID":"1b15d641-1837-462a-8743-661c7871f95b","Type":"ContainerStarted","Data":"a28e626707f3fb86f3535b54ae661f69761df58381755b0f182283dde12ac813"} Nov 01 00:52:12.980365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:12.980338 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" event={"ID":"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a","Type":"ContainerStarted","Data":"8a015a261e003cc010d36ee86fc3840c6229cae8d51c7fcdf58b93ff7d153e51"} Nov 01 00:52:13.101683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:13.100970 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" podStartSLOduration=2.100951974 podStartE2EDuration="2.100951974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:13.098896171 +0000 UTC m=+1723.078478503" watchObservedRunningTime="2026-11-01 00:52:13.100951974 +0000 UTC m=+1723.080534311" Nov 01 00:52:13.108871 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:52:13.108833 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:52:14.025997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:14.011522 4910 generic.go:356] "Generic (PLEG): container finished" podID="1b15d641-1837-462a-8743-661c7871f95b" containerID="29abb3aa0e916e54751725a73f5bc57b7f9d5af6c0575ae5a1dcaa34ccdccc96" exitCode=0 Nov 01 00:52:14.025997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:14.011753 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" event={"ID":"1b15d641-1837-462a-8743-661c7871f95b","Type":"ContainerDied","Data":"29abb3aa0e916e54751725a73f5bc57b7f9d5af6c0575ae5a1dcaa34ccdccc96"} Nov 01 00:52:14.025997 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:14.014049 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" event={"ID":"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a","Type":"ContainerStarted","Data":"4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f"} Nov 01 00:52:14.286864 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:14.286742 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podStartSLOduration=2.286723327 podStartE2EDuration="2.286723327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:14.262239815 +0000 UTC m=+1724.241822152" watchObservedRunningTime="2026-11-01 00:52:14.286723327 +0000 UTC m=+1724.266305661" Nov 01 00:52:14.503353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:14.503318 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:14.504802 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:14.504774 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:14.504930 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:14.504828 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:15.307294 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.307264 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:15.343713 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.343682 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b15d641-1837-462a-8743-661c7871f95b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15d641-1837-462a-8743-661c7871f95b-maas-api-xks-ca\") pod \"1b15d641-1837-462a-8743-661c7871f95b\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " Nov 01 00:52:15.343916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.343901 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b15d641-1837-462a-8743-661c7871f95b-kube-api-access-ktkw8\" (UniqueName: \"kubernetes.io/projected/1b15d641-1837-462a-8743-661c7871f95b-kube-api-access-ktkw8\") pod \"1b15d641-1837-462a-8743-661c7871f95b\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " Nov 01 00:52:15.344035 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.343930 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b15d641-1837-462a-8743-661c7871f95b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15d641-1837-462a-8743-661c7871f95b-maas-api-openshift-ca\") pod \"1b15d641-1837-462a-8743-661c7871f95b\" (UID: \"1b15d641-1837-462a-8743-661c7871f95b\") " Nov 01 00:52:15.344333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.344311 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b15d641-1837-462a-8743-661c7871f95b-maas-api-openshift-ca" pod "1b15d641-1837-462a-8743-661c7871f95b" (UID: "1b15d641-1837-462a-8743-661c7871f95b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:52:15.344515 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.344499 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15d641-1837-462a-8743-661c7871f95b-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:15.346243 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.346221 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b15d641-1837-462a-8743-661c7871f95b-kube-api-access-ktkw8" pod "1b15d641-1837-462a-8743-661c7871f95b" (UID: "1b15d641-1837-462a-8743-661c7871f95b"). InnerVolumeSpecName "kube-api-access-ktkw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:52:15.346505 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.346490 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b15d641-1837-462a-8743-661c7871f95b-maas-api-xks-ca" pod "1b15d641-1837-462a-8743-661c7871f95b" (UID: "1b15d641-1837-462a-8743-661c7871f95b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:52:15.446695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.446654 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkw8\" (UniqueName: \"kubernetes.io/projected/1b15d641-1837-462a-8743-661c7871f95b-kube-api-access-ktkw8\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:15.446695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.446694 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15d641-1837-462a-8743-661c7871f95b-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:15.503597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.503561 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:15.503857 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:15.503642 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:16.036733 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:16.036703 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" Nov 01 00:52:16.036919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:16.036736 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5" event={"ID":"1b15d641-1837-462a-8743-661c7871f95b","Type":"ContainerDied","Data":"a28e626707f3fb86f3535b54ae661f69761df58381755b0f182283dde12ac813"} Nov 01 00:52:16.036919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:16.036805 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a28e626707f3fb86f3535b54ae661f69761df58381755b0f182283dde12ac813" Nov 01 00:52:16.504084 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:16.504043 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:16.504481 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:16.504120 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:17.503650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:17.503604 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:17.503973 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:17.503678 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:18.503961 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:18.503930 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:18.504407 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:18.503996 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:19.503854 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:19.503820 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:19.504034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:19.503880 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:20.504567 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:20.504526 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:20.505051 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:20.504585 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:21.506803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:21.506767 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 00:52:21.507783 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:21.507741 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 00:52:23.511651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:23.509712 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:52:23.511651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:23.509787 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:23.756123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:23.756076 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:23.757731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:23.757701 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:23.770996 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:23.770920 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 00:52:23.840650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:23.836732 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:24.016789 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:24.016746 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:24.031785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:24.031697 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:24.745751 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:24.745697 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:28.542862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:28.542813 4910 patch_prober.go:29] interesting pod/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:52:28.543405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:28.542905 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:52:30.761375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:30.761333 4910 scope.go:122] "RemoveContainer" containerID="2580574e00cbffeb0add5084305073a6ce705ef51612f55b459544967280131c" Nov 01 00:52:30.776275 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:30.776246 4910 scope.go:122] "RemoveContainer" containerID="2290f209239740745607e753d50235bd0ffa98a2be3130b43117e1682d39761e" Nov 01 00:52:30.790143 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:30.790120 4910 scope.go:122] "RemoveContainer" containerID="58d965542eb45d2a006aabfa91004e48c24f111f122273ba13aedae4bcc808d2" Nov 01 00:52:30.801531 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:30.801506 4910 scope.go:122] "RemoveContainer" containerID="653d62081f55017950a834f0c8e06ad34526c6a4f5fa20e725127e809964a7ea" Nov 01 00:52:30.814916 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:30.814895 4910 scope.go:122] "RemoveContainer" containerID="d9a32ab3cad582dfa68da5a1bdc34f116d8fe96b8e4abc90e57362ea2c3f5f74" Nov 01 00:52:30.826012 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:30.825990 4910 scope.go:122] "RemoveContainer" containerID="1b19a2d0aaf0c4d3a4638feea09fe010a3c7f2861d4de04530226f9278559cc4" Nov 01 00:52:31.659936 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:31.659904 4910 generic.go:356] "Generic (PLEG): container finished" podID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerID="f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909" exitCode=0 Nov 01 00:52:31.660155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:31.659981 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" event={"ID":"69ddff29-fa3e-49e0-b6f4-71c62187e64f","Type":"ContainerDied","Data":"f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909"} Nov 01 00:52:31.660502 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:31.660482 4910 scope.go:122] "RemoveContainer" containerID="f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909" Nov 01 00:52:31.662435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:31.662390 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:31.662549 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:31.662483 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:31.662549 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:31.662533 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:52:32.387986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.387949 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg"] Nov 01 00:52:32.389136 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.389114 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1b15d641-1837-462a-8743-661c7871f95b" containerName="revoke-keys" Nov 01 00:52:32.389232 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.389139 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b15d641-1837-462a-8743-661c7871f95b" containerName="revoke-keys" Nov 01 00:52:32.389294 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.389283 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b15d641-1837-462a-8743-661c7871f95b" containerName="revoke-keys" Nov 01 00:52:32.393956 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.393926 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.431648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.430674 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kbw\" (UniqueName: \"kubernetes.io/projected/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-kube-api-access-d4kbw\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.431648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.430777 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.431648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.430845 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.450352 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.450254 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg"] Nov 01 00:52:32.516604 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.516575 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:52:32.516802 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.516745 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:52:32.532386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.532347 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.532547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.532433 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.532798 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.532773 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kbw\" (UniqueName: \"kubernetes.io/projected/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-kube-api-access-d4kbw\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.533081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.533065 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.535233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.535206 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.594866 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.594831 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kbw\" (UniqueName: \"kubernetes.io/projected/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-kube-api-access-d4kbw\") pod \"maas-api-revoke-sub-92cc638f-jlhjg\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.691160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.691071 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" event={"ID":"69ddff29-fa3e-49e0-b6f4-71c62187e64f","Type":"ContainerStarted","Data":"86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580"} Nov 01 00:52:32.711216 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.711191 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:32.941762 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.941667 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:52:32.941944 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.941908 4910 patch_prober.go:29] interesting pod/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 00:52:32.942017 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:32.941960 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 00:52:33.146940 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:52:33.146910 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5f549b_4bf9_495e_9bcb_3592c0b7be83.slice/crio-262f0544133a8d63b55a500500ba39eac68d7a43f29b4538eb0d8509ec70deb2 WatchSource:0}: Error finding container 262f0544133a8d63b55a500500ba39eac68d7a43f29b4538eb0d8509ec70deb2: Status 404 returned error can't find the container with id 262f0544133a8d63b55a500500ba39eac68d7a43f29b4538eb0d8509ec70deb2 Nov 01 00:52:33.149479 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:33.149455 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg"] Nov 01 00:52:33.704008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:33.703969 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" event={"ID":"5c5f549b-4bf9-495e-9bcb-3592c0b7be83","Type":"ContainerStarted","Data":"08aa0a83e8074c773a6ab59b9427dd5972eea06d86f960a5f502eade3eed7240"} Nov 01 00:52:33.704458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:33.704038 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" event={"ID":"5c5f549b-4bf9-495e-9bcb-3592c0b7be83","Type":"ContainerStarted","Data":"262f0544133a8d63b55a500500ba39eac68d7a43f29b4538eb0d8509ec70deb2"} Nov 01 00:52:33.826562 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:33.826511 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" podStartSLOduration=1.826497508 podStartE2EDuration="1.826497508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:52:33.821482846 +0000 UTC m=+1743.801065181" watchObservedRunningTime="2026-11-01 00:52:33.826497508 +0000 UTC m=+1743.806079843" Nov 01 00:52:33.941834 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:33.941754 4910 patch_prober.go:29] interesting pod/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 00:52:33.941834 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:33.941806 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 00:52:34.714735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:34.714648 4910 generic.go:356] "Generic (PLEG): container finished" podID="5c5f549b-4bf9-495e-9bcb-3592c0b7be83" containerID="08aa0a83e8074c773a6ab59b9427dd5972eea06d86f960a5f502eade3eed7240" exitCode=0 Nov 01 00:52:34.715159 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:34.714789 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" event={"ID":"5c5f549b-4bf9-495e-9bcb-3592c0b7be83","Type":"ContainerDied","Data":"08aa0a83e8074c773a6ab59b9427dd5972eea06d86f960a5f502eade3eed7240"} Nov 01 00:52:34.941768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:34.941740 4910 patch_prober.go:29] interesting pod/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 00:52:34.941921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:34.941789 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 00:52:35.942036 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:35.941975 4910 patch_prober.go:29] interesting pod/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 00:52:35.942417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:35.942091 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 00:52:35.995779 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:35.995719 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:36.130800 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.130745 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-xks-ca\") pod \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " Nov 01 00:52:36.131295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.131273 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-openshift-ca\") pod \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " Nov 01 00:52:36.131506 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.131457 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-kube-api-access-d4kbw\" (UniqueName: \"kubernetes.io/projected/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-kube-api-access-d4kbw\") pod \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\" (UID: \"5c5f549b-4bf9-495e-9bcb-3592c0b7be83\") " Nov 01 00:52:36.131898 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.131874 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-openshift-ca" pod "5c5f549b-4bf9-495e-9bcb-3592c0b7be83" (UID: "5c5f549b-4bf9-495e-9bcb-3592c0b7be83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:52:36.132419 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.132399 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:36.133638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.133581 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-xks-ca" pod "5c5f549b-4bf9-495e-9bcb-3592c0b7be83" (UID: "5c5f549b-4bf9-495e-9bcb-3592c0b7be83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:52:36.138749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.138726 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-kube-api-access-d4kbw" pod "5c5f549b-4bf9-495e-9bcb-3592c0b7be83" (UID: "5c5f549b-4bf9-495e-9bcb-3592c0b7be83"). InnerVolumeSpecName "kube-api-access-d4kbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:52:36.233593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.233564 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kbw\" (UniqueName: \"kubernetes.io/projected/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-kube-api-access-d4kbw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:36.233593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.233592 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5f549b-4bf9-495e-9bcb-3592c0b7be83-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:52:36.735428 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.735392 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" event={"ID":"5c5f549b-4bf9-495e-9bcb-3592c0b7be83","Type":"ContainerDied","Data":"262f0544133a8d63b55a500500ba39eac68d7a43f29b4538eb0d8509ec70deb2"} Nov 01 00:52:36.735606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.735440 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262f0544133a8d63b55a500500ba39eac68d7a43f29b4538eb0d8509ec70deb2" Nov 01 00:52:36.735606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.735446 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg" Nov 01 00:52:36.941675 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.941646 4910 patch_prober.go:29] interesting pod/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 00:52:36.941848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:36.941704 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 00:52:37.941765 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:37.941731 4910 patch_prober.go:29] interesting pod/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 01 00:52:37.942167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:37.941799 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 01 00:52:38.946195 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:38.946114 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:52:38.946708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:38.946518 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:52:38.947167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:38.947151 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:52:43.104651 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:52:43.104584 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:52:59.795651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:52:59.794895 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.016780 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.016740 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 00:53:00.401874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.400694 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.473764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.473711 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.652290 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.650714 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.741654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.705088 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.797609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.796886 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.797609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.797379 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:53:00.822316 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.822279 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:00.863373 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:00.861890 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 00:53:01.044068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.025362 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:01.044068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.037585 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:53:01.051835 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.049722 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:01.407707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.406979 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:01.424655 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.413302 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:53:01.481852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.481815 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:01.482763 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.482741 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:53:01.760651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.750033 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:01.959955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:01.959921 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:53:02.027444 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.027343 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 01 00:53:02.244383 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.244337 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:02.462674 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.458590 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:02.604684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.599570 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 00:53:02.742722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.727824 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.58:50052\" within 1s: context deadline exceeded" Nov 01 00:53:02.798655 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.780865 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:53:02.839331 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:53:02.839303 4910 logging.go:55] [core] [Channel #1586 SubChannel #1587]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.58:50052", ServerName: "10.132.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:44062->10.132.0.58:50052: use of closed network connection" Nov 01 00:53:02.846565 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.840408 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:53:02.880757 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.880709 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:02.961943 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:02.961899 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:03.091650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:03.050852 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:03.221510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:03.221457 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:53:03.229678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:03.229653 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 00:53:03.414748 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:03.414663 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 00:53:03.438917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:03.438892 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 00:53:03.495187 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:03.495161 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 00:53:03.513953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:03.513874 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 00:53:05.067975 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:05.067940 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6"] Nov 01 00:53:05.120453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:05.120416 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304bd02a-fkkk6"] Nov 01 00:53:06.196454 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.196419 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682b0f69-29fc-4a05-a330-9c24b4371708" path="/var/lib/kubelet/pods/682b0f69-29fc-4a05-a330-9c24b4371708/volumes" Nov 01 00:53:06.772633 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.772586 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt"] Nov 01 00:53:06.773431 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.773411 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5c5f549b-4bf9-495e-9bcb-3592c0b7be83" containerName="revoke-keys" Nov 01 00:53:06.773484 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.773433 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5f549b-4bf9-495e-9bcb-3592c0b7be83" containerName="revoke-keys" Nov 01 00:53:06.773565 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.773556 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5f549b-4bf9-495e-9bcb-3592c0b7be83" containerName="revoke-keys" Nov 01 00:53:06.778710 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.778684 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.836489 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.836457 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt"] Nov 01 00:53:06.878992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.878942 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:53:06.879184 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.879012 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:53:06.885945 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.885914 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.886118 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.886034 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvl9\" (UniqueName: \"kubernetes.io/projected/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-kube-api-access-sdvl9\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.886295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.886279 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.988316 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.988280 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.988496 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.988470 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.988558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.988526 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvl9\" (UniqueName: \"kubernetes.io/projected/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-kube-api-access-sdvl9\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.989111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.989085 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:06.991468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:06.991439 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:07.050333 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:07.050254 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvl9\" (UniqueName: \"kubernetes.io/projected/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-kube-api-access-sdvl9\") pod \"maas-api-revoke-keys-e2e-ait-64566096-fm9pt\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:07.095054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:07.095019 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:07.524011 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:53:07.523976 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a8befb_9a38_4e7e_b2b4_9e8d5700e973.slice/crio-eaa1067fcdfd71fd3f5b6c0da91c8c8f3ec4db41d7929427d8e4ad623f294b63 WatchSource:0}: Error finding container eaa1067fcdfd71fd3f5b6c0da91c8c8f3ec4db41d7929427d8e4ad623f294b63: Status 404 returned error can't find the container with id eaa1067fcdfd71fd3f5b6c0da91c8c8f3ec4db41d7929427d8e4ad623f294b63 Nov 01 00:53:07.524591 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:07.524564 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt"] Nov 01 00:53:08.274169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:08.274127 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" event={"ID":"83a8befb-9a38-4e7e-b2b4-9e8d5700e973","Type":"ContainerStarted","Data":"1f57f592aaba80dc0387d07bae6df74f15b046d04e46008ace1b067e6d3e2fc9"} Nov 01 00:53:08.274399 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:08.274203 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" event={"ID":"83a8befb-9a38-4e7e-b2b4-9e8d5700e973","Type":"ContainerStarted","Data":"eaa1067fcdfd71fd3f5b6c0da91c8c8f3ec4db41d7929427d8e4ad623f294b63"} Nov 01 00:53:08.393900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:08.393844 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" podStartSLOduration=2.393829777 podStartE2EDuration="2.393829777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:53:08.392230005 +0000 UTC m=+1778.371812360" watchObservedRunningTime="2026-11-01 00:53:08.393829777 +0000 UTC m=+1778.373412110" Nov 01 00:53:09.286055 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.285964 4910 generic.go:356] "Generic (PLEG): container finished" podID="c15ef169-84ef-4678-b219-641efe029c30" containerID="ba74432c7e52ec425f48319ca19f8b0191c139e92d54b7adc5c107486af8c727" exitCode=0 Nov 01 00:53:09.286055 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.286039 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" event={"ID":"c15ef169-84ef-4678-b219-641efe029c30","Type":"ContainerDied","Data":"ba74432c7e52ec425f48319ca19f8b0191c139e92d54b7adc5c107486af8c727"} Nov 01 00:53:09.286680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.286513 4910 scope.go:122] "RemoveContainer" containerID="ba74432c7e52ec425f48319ca19f8b0191c139e92d54b7adc5c107486af8c727" Nov 01 00:53:09.288372 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.288338 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:53:09.288503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.288377 4910 generic.go:356] "Generic (PLEG): container finished" podID="83a8befb-9a38-4e7e-b2b4-9e8d5700e973" containerID="1f57f592aaba80dc0387d07bae6df74f15b046d04e46008ace1b067e6d3e2fc9" exitCode=0 Nov 01 00:53:09.288503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.288407 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" event={"ID":"83a8befb-9a38-4e7e-b2b4-9e8d5700e973","Type":"ContainerDied","Data":"1f57f592aaba80dc0387d07bae6df74f15b046d04e46008ace1b067e6d3e2fc9"} Nov 01 00:53:09.288503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.288435 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:53:09.288503 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:09.288488 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:53:10.300704 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.300665 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" event={"ID":"c15ef169-84ef-4678-b219-641efe029c30","Type":"ContainerStarted","Data":"8b9127789e730cb1acd5afcc6184ca77b2471a84106432c590d8d91413989984"} Nov 01 00:53:10.531556 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.531531 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:10.669699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.665254 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-openshift-ca\") pod \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " Nov 01 00:53:10.669699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.665922 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-xks-ca\") pod \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " Nov 01 00:53:10.669699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.666075 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-kube-api-access-sdvl9\" (UniqueName: \"kubernetes.io/projected/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-kube-api-access-sdvl9\") pod \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\" (UID: \"83a8befb-9a38-4e7e-b2b4-9e8d5700e973\") " Nov 01 00:53:10.669699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.665706 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-openshift-ca" pod "83a8befb-9a38-4e7e-b2b4-9e8d5700e973" (UID: "83a8befb-9a38-4e7e-b2b4-9e8d5700e973"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:10.683237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.682905 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-xks-ca" pod "83a8befb-9a38-4e7e-b2b4-9e8d5700e973" (UID: "83a8befb-9a38-4e7e-b2b4-9e8d5700e973"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:53:10.689929 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.689893 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-kube-api-access-sdvl9" pod "83a8befb-9a38-4e7e-b2b4-9e8d5700e973" (UID: "83a8befb-9a38-4e7e-b2b4-9e8d5700e973"). InnerVolumeSpecName "kube-api-access-sdvl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:10.780653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.776056 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:10.780653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.776086 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:10.780653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:10.776096 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvl9\" (UniqueName: \"kubernetes.io/projected/83a8befb-9a38-4e7e-b2b4-9e8d5700e973-kube-api-access-sdvl9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:11.215145 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:11.215110 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:53:11.226153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:11.226127 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:53:11.227135 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:11.227115 4910 patch_prober.go:29] interesting pod/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 00:53:11.227210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:11.227165 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 00:53:11.311775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:11.311740 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" Nov 01 00:53:11.312173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:11.311783 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt" event={"ID":"83a8befb-9a38-4e7e-b2b4-9e8d5700e973","Type":"ContainerDied","Data":"eaa1067fcdfd71fd3f5b6c0da91c8c8f3ec4db41d7929427d8e4ad623f294b63"} Nov 01 00:53:11.312173 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:11.311839 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaa1067fcdfd71fd3f5b6c0da91c8c8f3ec4db41d7929427d8e4ad623f294b63" Nov 01 00:53:12.226966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.226932 4910 patch_prober.go:29] interesting pod/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 00:53:12.227151 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.227002 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 00:53:12.685104 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.685069 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb"] Nov 01 00:53:12.685891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.685872 4910 cpu_manager.go:414] "containerMap: removing container" podUID="83a8befb-9a38-4e7e-b2b4-9e8d5700e973" containerName="revoke-keys" Nov 01 00:53:12.685946 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.685895 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a8befb-9a38-4e7e-b2b4-9e8d5700e973" containerName="revoke-keys" Nov 01 00:53:12.686145 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.686132 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a8befb-9a38-4e7e-b2b4-9e8d5700e973" containerName="revoke-keys" Nov 01 00:53:12.690422 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.690404 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.794953 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.794914 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb"] Nov 01 00:53:12.807516 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.807427 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zgtk\" (UniqueName: \"kubernetes.io/projected/71391fc9-311d-4c55-a5f7-ba1403b87a55-kube-api-access-7zgtk\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.807516 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.807496 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.807881 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.807853 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.809267 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.809246 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:53:12.809410 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.809263 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:53:12.909438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.909410 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.909600 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.909519 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zgtk\" (UniqueName: \"kubernetes.io/projected/71391fc9-311d-4c55-a5f7-ba1403b87a55-kube-api-access-7zgtk\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.909600 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.909549 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.910079 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.910058 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.911859 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.911837 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:12.977011 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:12.976930 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zgtk\" (UniqueName: \"kubernetes.io/projected/71391fc9-311d-4c55-a5f7-ba1403b87a55-kube-api-access-7zgtk\") pod \"maas-api-revoke-keys-e2e-ait-8a74f100-n52qb\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:13.005416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:13.005390 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:13.130670 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:53:13.130387 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:53:13.226571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:13.226532 4910 patch_prober.go:29] interesting pod/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 00:53:13.226760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:13.226612 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 00:53:13.435433 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:53:13.435405 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71391fc9_311d_4c55_a5f7_ba1403b87a55.slice/crio-d86bb8d7f245de50effde7c8daa28f066f448d11915fe200a55986c401535042 WatchSource:0}: Error finding container d86bb8d7f245de50effde7c8daa28f066f448d11915fe200a55986c401535042: Status 404 returned error can't find the container with id d86bb8d7f245de50effde7c8daa28f066f448d11915fe200a55986c401535042 Nov 01 00:53:13.437362 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:13.437339 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb"] Nov 01 00:53:14.146934 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.146898 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc"] Nov 01 00:53:14.179650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.178954 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.226308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.226276 4910 patch_prober.go:29] interesting pod/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 00:53:14.226455 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.226337 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 00:53:14.230103 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.230078 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.230252 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.230234 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjzv\" (UniqueName: \"kubernetes.io/projected/325a6818-cedf-4573-b3fe-843afefbca0d-kube-api-access-ckjzv\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.230368 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.230354 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.307035 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.306234 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc"] Nov 01 00:53:14.333499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.332929 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjzv\" (UniqueName: \"kubernetes.io/projected/325a6818-cedf-4573-b3fe-843afefbca0d-kube-api-access-ckjzv\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.333692 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.333663 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.333856 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.333836 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.334460 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.334438 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.336769 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.336747 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.342099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.342075 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" event={"ID":"71391fc9-311d-4c55-a5f7-ba1403b87a55","Type":"ContainerStarted","Data":"5f241662c54750c2fa3609049f6cec54e0ff4c867138a6267028bc8e333c31e6"} Nov 01 00:53:14.342208 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.342123 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" event={"ID":"71391fc9-311d-4c55-a5f7-ba1403b87a55","Type":"ContainerStarted","Data":"d86bb8d7f245de50effde7c8daa28f066f448d11915fe200a55986c401535042"} Nov 01 00:53:14.400711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.399611 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjzv\" (UniqueName: \"kubernetes.io/projected/325a6818-cedf-4573-b3fe-843afefbca0d-kube-api-access-ckjzv\") pod \"maas-api-revoke-sub-ad05b88d-xvrbc\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.495738 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.495667 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" podStartSLOduration=2.495645386 podStartE2EDuration="2.495645386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:53:14.462852505 +0000 UTC m=+1784.442434844" watchObservedRunningTime="2026-11-01 00:53:14.495645386 +0000 UTC m=+1784.475227719" Nov 01 00:53:14.500983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.500954 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:14.931923 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:53:14.931893 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325a6818_cedf_4573_b3fe_843afefbca0d.slice/crio-9fca92443aaa4a945f05e83dc254efad98878702f01e9874babf44e1747dd64d WatchSource:0}: Error finding container 9fca92443aaa4a945f05e83dc254efad98878702f01e9874babf44e1747dd64d: Status 404 returned error can't find the container with id 9fca92443aaa4a945f05e83dc254efad98878702f01e9874babf44e1747dd64d Nov 01 00:53:14.933457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:14.933429 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc"] Nov 01 00:53:15.226782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:15.226741 4910 patch_prober.go:29] interesting pod/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 01 00:53:15.227164 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:15.226804 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 01 00:53:15.357828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:15.357779 4910 generic.go:356] "Generic (PLEG): container finished" podID="71391fc9-311d-4c55-a5f7-ba1403b87a55" containerID="5f241662c54750c2fa3609049f6cec54e0ff4c867138a6267028bc8e333c31e6" exitCode=0 Nov 01 00:53:15.358010 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:15.357918 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" event={"ID":"71391fc9-311d-4c55-a5f7-ba1403b87a55","Type":"ContainerDied","Data":"5f241662c54750c2fa3609049f6cec54e0ff4c867138a6267028bc8e333c31e6"} Nov 01 00:53:15.360009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:15.359970 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" event={"ID":"325a6818-cedf-4573-b3fe-843afefbca0d","Type":"ContainerStarted","Data":"9fca92443aaa4a945f05e83dc254efad98878702f01e9874babf44e1747dd64d"} Nov 01 00:53:16.230813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.230738 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:53:16.232076 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.232055 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:53:16.372411 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.372378 4910 generic.go:356] "Generic (PLEG): container finished" podID="325a6818-cedf-4573-b3fe-843afefbca0d" containerID="fa246e0929a725bdc003b0ac25198833202dc47b4a46897f378210fc050fd951" exitCode=0 Nov 01 00:53:16.372591 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.372465 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" event={"ID":"325a6818-cedf-4573-b3fe-843afefbca0d","Type":"ContainerDied","Data":"fa246e0929a725bdc003b0ac25198833202dc47b4a46897f378210fc050fd951"} Nov 01 00:53:16.563800 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.563772 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:16.668068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.668041 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-openshift-ca\") pod \"71391fc9-311d-4c55-a5f7-ba1403b87a55\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " Nov 01 00:53:16.668247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.668148 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-xks-ca\") pod \"71391fc9-311d-4c55-a5f7-ba1403b87a55\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " Nov 01 00:53:16.668247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.668222 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71391fc9-311d-4c55-a5f7-ba1403b87a55-kube-api-access-7zgtk\" (UniqueName: \"kubernetes.io/projected/71391fc9-311d-4c55-a5f7-ba1403b87a55-kube-api-access-7zgtk\") pod \"71391fc9-311d-4c55-a5f7-ba1403b87a55\" (UID: \"71391fc9-311d-4c55-a5f7-ba1403b87a55\") " Nov 01 00:53:16.668450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.668429 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-openshift-ca" pod "71391fc9-311d-4c55-a5f7-ba1403b87a55" (UID: "71391fc9-311d-4c55-a5f7-ba1403b87a55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:16.669344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.669314 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:16.670584 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.670554 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-xks-ca" pod "71391fc9-311d-4c55-a5f7-ba1403b87a55" (UID: "71391fc9-311d-4c55-a5f7-ba1403b87a55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:53:16.670695 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.670562 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71391fc9-311d-4c55-a5f7-ba1403b87a55-kube-api-access-7zgtk" pod "71391fc9-311d-4c55-a5f7-ba1403b87a55" (UID: "71391fc9-311d-4c55-a5f7-ba1403b87a55"). InnerVolumeSpecName "kube-api-access-7zgtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:16.771078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.771041 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71391fc9-311d-4c55-a5f7-ba1403b87a55-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:16.771078 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:16.771078 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zgtk\" (UniqueName: \"kubernetes.io/projected/71391fc9-311d-4c55-a5f7-ba1403b87a55-kube-api-access-7zgtk\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:17.385676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.385615 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" Nov 01 00:53:17.385676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.385654 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb" event={"ID":"71391fc9-311d-4c55-a5f7-ba1403b87a55","Type":"ContainerDied","Data":"d86bb8d7f245de50effde7c8daa28f066f448d11915fe200a55986c401535042"} Nov 01 00:53:17.386123 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.385708 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d86bb8d7f245de50effde7c8daa28f066f448d11915fe200a55986c401535042" Nov 01 00:53:17.579155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.579129 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:17.683811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.683729 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-openshift-ca\") pod \"325a6818-cedf-4573-b3fe-843afefbca0d\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " Nov 01 00:53:17.683971 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.683919 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-xks-ca\") pod \"325a6818-cedf-4573-b3fe-843afefbca0d\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " Nov 01 00:53:17.684054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.684038 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325a6818-cedf-4573-b3fe-843afefbca0d-kube-api-access-ckjzv\" (UniqueName: \"kubernetes.io/projected/325a6818-cedf-4573-b3fe-843afefbca0d-kube-api-access-ckjzv\") pod \"325a6818-cedf-4573-b3fe-843afefbca0d\" (UID: \"325a6818-cedf-4573-b3fe-843afefbca0d\") " Nov 01 00:53:17.684192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.684167 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-openshift-ca" pod "325a6818-cedf-4573-b3fe-843afefbca0d" (UID: "325a6818-cedf-4573-b3fe-843afefbca0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:17.684882 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.684814 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:17.686217 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.686185 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-xks-ca" pod "325a6818-cedf-4573-b3fe-843afefbca0d" (UID: "325a6818-cedf-4573-b3fe-843afefbca0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:53:17.686415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.686392 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325a6818-cedf-4573-b3fe-843afefbca0d-kube-api-access-ckjzv" pod "325a6818-cedf-4573-b3fe-843afefbca0d" (UID: "325a6818-cedf-4573-b3fe-843afefbca0d"). InnerVolumeSpecName "kube-api-access-ckjzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:17.786547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.786500 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a6818-cedf-4573-b3fe-843afefbca0d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:17.786547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:17.786541 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckjzv\" (UniqueName: \"kubernetes.io/projected/325a6818-cedf-4573-b3fe-843afefbca0d-kube-api-access-ckjzv\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:18.400555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:18.400525 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" Nov 01 00:53:18.400555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:18.400536 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc" event={"ID":"325a6818-cedf-4573-b3fe-843afefbca0d","Type":"ContainerDied","Data":"9fca92443aaa4a945f05e83dc254efad98878702f01e9874babf44e1747dd64d"} Nov 01 00:53:18.401014 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:18.400590 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fca92443aaa4a945f05e83dc254efad98878702f01e9874babf44e1747dd64d" Nov 01 00:53:30.597033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:30.597000 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:53:30.599452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:30.599428 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:53:30.611286 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:30.611249 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:53:30.613897 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:30.613879 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:53:36.050700 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:36.050664 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc"] Nov 01 00:53:36.100511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:36.100477 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974ade6e-952bc"] Nov 01 00:53:36.195508 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:36.195472 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c850074f-db5e-4858-9d41-e03da64160c9" path="/var/lib/kubelet/pods/c850074f-db5e-4858-9d41-e03da64160c9/volumes" Nov 01 00:53:43.073124 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.073013 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl"] Nov 01 00:53:43.074954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.074926 4910 cpu_manager.go:414] "containerMap: removing container" podUID="71391fc9-311d-4c55-a5f7-ba1403b87a55" containerName="revoke-keys" Nov 01 00:53:43.074954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.074956 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71391fc9-311d-4c55-a5f7-ba1403b87a55" containerName="revoke-keys" Nov 01 00:53:43.075164 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.074991 4910 cpu_manager.go:414] "containerMap: removing container" podUID="325a6818-cedf-4573-b3fe-843afefbca0d" containerName="revoke-keys" Nov 01 00:53:43.075164 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.074999 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325a6818-cedf-4573-b3fe-843afefbca0d" containerName="revoke-keys" Nov 01 00:53:43.075427 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.075405 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="71391fc9-311d-4c55-a5f7-ba1403b87a55" containerName="revoke-keys" Nov 01 00:53:43.075508 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.075435 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="325a6818-cedf-4573-b3fe-843afefbca0d" containerName="revoke-keys" Nov 01 00:53:43.084385 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.084365 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.118909 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:53:43.118876 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:53:43.125688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.125661 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt42j\" (UniqueName: \"kubernetes.io/projected/68f401da-6ed5-4b8a-a098-cb189b776803-kube-api-access-qt42j\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.125847 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.125796 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.125913 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.125876 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.130434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.130411 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl"] Nov 01 00:53:43.176370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.170833 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:53:43.176370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.170988 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:53:43.227837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.227078 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt42j\" (UniqueName: \"kubernetes.io/projected/68f401da-6ed5-4b8a-a098-cb189b776803-kube-api-access-qt42j\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.227837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.227161 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.227837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.227233 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.228268 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.228238 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.231416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.231371 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.290850 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.290818 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt42j\" (UniqueName: \"kubernetes.io/projected/68f401da-6ed5-4b8a-a098-cb189b776803-kube-api-access-qt42j\") pod \"maas-api-revoke-keys-e2e-migrate-14957a74-twxgl\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.416812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.416725 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:43.873644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:43.872598 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl"] Nov 01 00:53:44.683511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:44.683411 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" event={"ID":"68f401da-6ed5-4b8a-a098-cb189b776803","Type":"ContainerStarted","Data":"8e2794558d3e6b6626e1f75a119f0c2bd57c197750753723523871816705712c"} Nov 01 00:53:44.683511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:44.683484 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" event={"ID":"68f401da-6ed5-4b8a-a098-cb189b776803","Type":"ContainerStarted","Data":"3687365beb0dd432e495ba7ce1b3b48929e266f926c9c4436b05e2c01d76beca"} Nov 01 00:53:44.821412 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:44.821361 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" podStartSLOduration=2.821343055 podStartE2EDuration="2.821343055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:53:44.801518864 +0000 UTC m=+1814.781101214" watchObservedRunningTime="2026-11-01 00:53:44.821343055 +0000 UTC m=+1814.800925390" Nov 01 00:53:45.695407 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:45.695368 4910 generic.go:356] "Generic (PLEG): container finished" podID="68f401da-6ed5-4b8a-a098-cb189b776803" containerID="8e2794558d3e6b6626e1f75a119f0c2bd57c197750753723523871816705712c" exitCode=0 Nov 01 00:53:45.695845 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:45.695447 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" event={"ID":"68f401da-6ed5-4b8a-a098-cb189b776803","Type":"ContainerDied","Data":"8e2794558d3e6b6626e1f75a119f0c2bd57c197750753723523871816705712c"} Nov 01 00:53:46.872457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:46.872436 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:47.010607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.010566 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-xks-ca\") pod \"68f401da-6ed5-4b8a-a098-cb189b776803\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " Nov 01 00:53:47.010787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.010770 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f401da-6ed5-4b8a-a098-cb189b776803-kube-api-access-qt42j\" (UniqueName: \"kubernetes.io/projected/68f401da-6ed5-4b8a-a098-cb189b776803-kube-api-access-qt42j\") pod \"68f401da-6ed5-4b8a-a098-cb189b776803\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " Nov 01 00:53:47.010954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.010936 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-openshift-ca\") pod \"68f401da-6ed5-4b8a-a098-cb189b776803\" (UID: \"68f401da-6ed5-4b8a-a098-cb189b776803\") " Nov 01 00:53:47.011362 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.011331 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-openshift-ca" pod "68f401da-6ed5-4b8a-a098-cb189b776803" (UID: "68f401da-6ed5-4b8a-a098-cb189b776803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:53:47.011713 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.011690 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:47.013335 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.013307 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f401da-6ed5-4b8a-a098-cb189b776803-kube-api-access-qt42j" pod "68f401da-6ed5-4b8a-a098-cb189b776803" (UID: "68f401da-6ed5-4b8a-a098-cb189b776803"). InnerVolumeSpecName "kube-api-access-qt42j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:53:47.013435 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.013412 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-xks-ca" pod "68f401da-6ed5-4b8a-a098-cb189b776803" (UID: "68f401da-6ed5-4b8a-a098-cb189b776803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:53:47.112709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.112680 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f401da-6ed5-4b8a-a098-cb189b776803-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:47.112709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.112705 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt42j\" (UniqueName: \"kubernetes.io/projected/68f401da-6ed5-4b8a-a098-cb189b776803-kube-api-access-qt42j\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:53:47.717416 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.717391 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" Nov 01 00:53:47.717739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.717416 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl" event={"ID":"68f401da-6ed5-4b8a-a098-cb189b776803","Type":"ContainerDied","Data":"3687365beb0dd432e495ba7ce1b3b48929e266f926c9c4436b05e2c01d76beca"} Nov 01 00:53:47.717739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:47.717463 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3687365beb0dd432e495ba7ce1b3b48929e266f926c9c4436b05e2c01d76beca" Nov 01 00:53:51.053856 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:51.053825 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx"] Nov 01 00:53:51.103446 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:51.103412 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9efe04b8-nwttx"] Nov 01 00:53:52.219917 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:52.219879 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae290517-e58a-48ff-8e28-495f3ce917e3" path="/var/lib/kubelet/pods/ae290517-e58a-48ff-8e28-495f3ce917e3/volumes" Nov 01 00:53:54.140648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:54.134156 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt"] Nov 01 00:53:54.167528 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:54.167479 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64566096-fm9pt"] Nov 01 00:53:54.326155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:54.326119 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a8befb-9a38-4e7e-b2b4-9e8d5700e973" path="/var/lib/kubelet/pods/83a8befb-9a38-4e7e-b2b4-9e8d5700e973/volumes" Nov 01 00:53:55.571591 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:55.571558 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb"] Nov 01 00:53:55.621323 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:55.621284 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a74f100-n52qb"] Nov 01 00:53:56.131570 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:56.131535 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5"] Nov 01 00:53:56.131873 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:56.131843 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" containerID="cri-o://86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580" gracePeriod=30 Nov 01 00:53:56.220109 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:53:56.220071 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71391fc9-311d-4c55-a5f7-ba1403b87a55" path="/var/lib/kubelet/pods/71391fc9-311d-4c55-a5f7-ba1403b87a55/volumes" Nov 01 00:54:01.467615 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.467586 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:54:01.510557 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.510528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-kube-api-access-zhmgf\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-kube-api-access-zhmgf\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.510760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.510573 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istiod-ca-cert\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.510760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.510672 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-envoy\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.510760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.510737 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-certs\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.510935 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.510804 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-podinfo\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.511057 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511036 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-data\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.511162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511089 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-token\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-token\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.511221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511171 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-credential-socket\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.511221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511034 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-certs" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:01.511221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511215 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-socket\") pod \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\" (UID: \"69ddff29-fa3e-49e0-b6f4-71c62187e64f\") " Nov 01 00:54:01.511392 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511077 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istiod-ca-cert" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:54:01.511392 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511246 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-data" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:01.511679 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511610 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-credential-socket" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:01.511899 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.511737 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-socket" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:01.512029 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.512006 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.512099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.512037 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.512099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.512055 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.512099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.512070 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.512099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.512084 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.513142 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.513110 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-envoy" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:01.513237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.513140 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-kube-api-access-zhmgf" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "kube-api-access-zhmgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:54:01.513388 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.513365 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-token" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:54:01.513497 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.513438 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-podinfo" pod "69ddff29-fa3e-49e0-b6f4-71c62187e64f" (UID: "69ddff29-fa3e-49e0-b6f4-71c62187e64f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:54:01.614024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.613996 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.614024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.614022 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhmgf\" (UniqueName: \"kubernetes.io/projected/69ddff29-fa3e-49e0-b6f4-71c62187e64f-kube-api-access-zhmgf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.614024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.614033 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:01.614264 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:01.614042 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ddff29-fa3e-49e0-b6f4-71c62187e64f-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:02.121938 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.121903 4910 generic.go:356] "Generic (PLEG): container finished" podID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerID="86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580" exitCode=0 Nov 01 00:54:02.122126 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.121990 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" Nov 01 00:54:02.122126 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.122019 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" event={"ID":"69ddff29-fa3e-49e0-b6f4-71c62187e64f","Type":"ContainerDied","Data":"86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580"} Nov 01 00:54:02.122126 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.122107 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5" event={"ID":"69ddff29-fa3e-49e0-b6f4-71c62187e64f","Type":"ContainerDied","Data":"b328090b230fe93f71be83b51ba9efc4e37c2936ef1df4a5736df7a12ac9a7d0"} Nov 01 00:54:02.122278 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.122136 4910 scope.go:122] "RemoveContainer" containerID="86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580" Nov 01 00:54:02.135177 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.135155 4910 scope.go:122] "RemoveContainer" containerID="f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909" Nov 01 00:54:02.146465 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.146441 4910 scope.go:122] "RemoveContainer" containerID="86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580" Nov 01 00:54:02.146806 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:54:02.146787 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580\": container with ID starting with 86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580 not found: ID does not exist" containerID="86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580" Nov 01 00:54:02.146914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.146813 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580"} err="failed to get container status \"86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580\": rpc error: code = NotFound desc = could not find container \"86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580\": container with ID starting with 86a1311a845cc7126a00d7f9064bd1e2649425b1e253b7a385443f6a0f9f4580 not found: ID does not exist" Nov 01 00:54:02.146914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.146831 4910 scope.go:122] "RemoveContainer" containerID="f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909" Nov 01 00:54:02.147106 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:54:02.147090 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909\": container with ID starting with f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909 not found: ID does not exist" containerID="f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909" Nov 01 00:54:02.147167 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.147110 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909"} err="failed to get container status \"f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909\": rpc error: code = NotFound desc = could not find container \"f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909\": container with ID starting with f2494792164d1600c2d347e3b5accccf395e44af1430c17114f8fe0eff9ad909 not found: ID does not exist" Nov 01 00:54:02.341816 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.341782 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5"] Nov 01 00:54:02.391348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:02.391278 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8a74f100-openshift-default-6777d99847-pw2q5"] Nov 01 00:54:04.260458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:04.260421 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" path="/var/lib/kubelet/pods/69ddff29-fa3e-49e0-b6f4-71c62187e64f/volumes" Nov 01 00:54:13.104673 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:54:13.104603 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:54:15.059571 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:15.059527 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2"] Nov 01 00:54:15.065113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:15.065089 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4cd57a-gdll2"] Nov 01 00:54:16.197837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:16.197806 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4e1361-e16e-4d9b-8f77-90ac550a6909" path="/var/lib/kubelet/pods/2f4e1361-e16e-4d9b-8f77-90ac550a6909/volumes" Nov 01 00:54:23.057589 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:23.057348 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl"] Nov 01 00:54:23.108579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:23.108546 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-14957a74-twxgl"] Nov 01 00:54:23.146094 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:23.146065 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78"] Nov 01 00:54:23.146651 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:23.146566 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" containerID="cri-o://8b9127789e730cb1acd5afcc6184ca77b2471a84106432c590d8d91413989984" gracePeriod=30 Nov 01 00:54:24.201404 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:24.201367 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f401da-6ed5-4b8a-a098-cb189b776803" path="/var/lib/kubelet/pods/68f401da-6ed5-4b8a-a098-cb189b776803/volumes" Nov 01 00:54:28.482896 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.482850 4910 generic.go:356] "Generic (PLEG): container finished" podID="c15ef169-84ef-4678-b219-641efe029c30" containerID="8b9127789e730cb1acd5afcc6184ca77b2471a84106432c590d8d91413989984" exitCode=0 Nov 01 00:54:28.483699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.482926 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" event={"ID":"c15ef169-84ef-4678-b219-641efe029c30","Type":"ContainerDied","Data":"8b9127789e730cb1acd5afcc6184ca77b2471a84106432c590d8d91413989984"} Nov 01 00:54:28.483699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.483009 4910 scope.go:122] "RemoveContainer" containerID="ba74432c7e52ec425f48319ca19f8b0191c139e92d54b7adc5c107486af8c727" Nov 01 00:54:28.500469 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.500435 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512220 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-envoy\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512305 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-data\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512381 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-istio-token\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-istio-token\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512452 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-credential-socket\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512510 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15ef169-84ef-4678-b219-641efe029c30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15ef169-84ef-4678-b219-641efe029c30-istiod-ca-cert\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512619 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-socket\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512663 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-kube-api-access-dbg5z\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-kube-api-access-dbg5z\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-certs\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.512759 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15ef169-84ef-4678-b219-641efe029c30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15ef169-84ef-4678-b219-641efe029c30-istio-podinfo\") pod \"c15ef169-84ef-4678-b219-641efe029c30\" (UID: \"c15ef169-84ef-4678-b219-641efe029c30\") " Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.513480 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-credential-socket" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.513568 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-data" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:28.513831 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.513769 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-certs" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:28.514568 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.513863 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15ef169-84ef-4678-b219-641efe029c30-istiod-ca-cert" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:54:28.514568 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.513909 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-socket" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:28.515417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.515159 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-envoy" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:54:28.516045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.516020 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-kube-api-access-dbg5z" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "kube-api-access-dbg5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:54:28.516309 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.516290 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-istio-token" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:54:28.518079 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.518055 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15ef169-84ef-4678-b219-641efe029c30-istio-podinfo" pod "c15ef169-84ef-4678-b219-641efe029c30" (UID: "c15ef169-84ef-4678-b219-641efe029c30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:54:28.614699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614671 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614700 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15ef169-84ef-4678-b219-641efe029c30-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614699 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614714 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614727 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbg5z\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-kube-api-access-dbg5z\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614740 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614752 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15ef169-84ef-4678-b219-641efe029c30-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614762 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614776 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15ef169-84ef-4678-b219-641efe029c30-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:28.614947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:28.614788 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15ef169-84ef-4678-b219-641efe029c30-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:54:29.494711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:29.494683 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" Nov 01 00:54:29.494711 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:29.494697 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78" event={"ID":"c15ef169-84ef-4678-b219-641efe029c30","Type":"ContainerDied","Data":"747f999719908f81ba6a4ff4fb3d5973023913774eaa39cf93713a919bfba10c"} Nov 01 00:54:29.495244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:29.494758 4910 scope.go:122] "RemoveContainer" containerID="8b9127789e730cb1acd5afcc6184ca77b2471a84106432c590d8d91413989984" Nov 01 00:54:29.736335 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:29.736300 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78"] Nov 01 00:54:29.741786 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:29.741764 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-14957a74-gw-openshift-default-c4dc7bd4f-fjv78"] Nov 01 00:54:30.200095 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:30.200055 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15ef169-84ef-4678-b219-641efe029c30" path="/var/lib/kubelet/pods/c15ef169-84ef-4678-b219-641efe029c30/volumes" Nov 01 00:54:30.960439 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:30.960411 4910 scope.go:122] "RemoveContainer" containerID="0bc30325aed8aae40f13eef7852342ef63e76e19cf1ec39c8e4cb38fa57f56c9" Nov 01 00:54:31.087952 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:31.087913 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km"] Nov 01 00:54:31.137783 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:31.137756 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112865ae-227km"] Nov 01 00:54:32.195072 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:32.195038 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d229ca-8be4-48ed-b6dd-e48e327a922c" path="/var/lib/kubelet/pods/60d229ca-8be4-48ed-b6dd-e48e327a922c/volumes" Nov 01 00:54:40.051970 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:40.051935 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv"] Nov 01 00:54:40.101782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:40.101745 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724a1cfd-s8lgv"] Nov 01 00:54:40.195906 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:40.195872 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff287dd-b20c-4bb7-adf1-37be58be6c7a" path="/var/lib/kubelet/pods/3ff287dd-b20c-4bb7-adf1-37be58be6c7a/volumes" Nov 01 00:54:43.115661 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:54:43.115585 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:54:51.053461 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:51.053418 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76"] Nov 01 00:54:51.107773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:51.107727 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91473b4d-97w76"] Nov 01 00:54:52.056526 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:52.056496 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj"] Nov 01 00:54:52.106656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:52.106571 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1821359e-9r8sj"] Nov 01 00:54:52.201096 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:52.201064 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594d72da-b160-4647-a3c6-8af7aae13eb2" path="/var/lib/kubelet/pods/594d72da-b160-4647-a3c6-8af7aae13eb2/volumes" Nov 01 00:54:52.201698 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:52.201681 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e707298e-d9a3-4d29-96f1-b15b5759a9d1" path="/var/lib/kubelet/pods/e707298e-d9a3-4d29-96f1-b15b5759a9d1/volumes" Nov 01 00:54:55.729390 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.729355 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn"] Nov 01 00:54:55.730107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730091 4910 cpu_manager.go:414] "containerMap: removing container" podUID="68f401da-6ed5-4b8a-a098-cb189b776803" containerName="revoke-keys" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730108 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f401da-6ed5-4b8a-a098-cb189b776803" containerName="revoke-keys" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730121 4910 cpu_manager.go:414] "containerMap: removing container" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730125 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730136 4910 cpu_manager.go:414] "containerMap: removing container" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730141 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730148 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730153 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" Nov 01 00:54:55.730157 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730159 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" Nov 01 00:54:55.730405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730164 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" Nov 01 00:54:55.730405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730286 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f401da-6ed5-4b8a-a098-cb189b776803" containerName="revoke-keys" Nov 01 00:54:55.730405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730307 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" Nov 01 00:54:55.730405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730314 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" Nov 01 00:54:55.730405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.730321 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15ef169-84ef-4678-b219-641efe029c30" containerName="istio-proxy" Nov 01 00:54:55.734295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.734277 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.804820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.804788 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a31bcdff-openshift-default-dockercfg-srwr6\"" Nov 01 00:54:55.820479 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820448 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-credential-socket\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.820680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820503 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-token\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.820680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820576 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-data\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.820680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820608 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-certs\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.820925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820716 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-envoy\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.820925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820824 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqfn\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-kube-api-access-xfqfn\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.820925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820885 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-socket\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.820925 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.820916 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istiod-ca-cert\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.821176 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.821052 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-podinfo\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.861041 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.861011 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn"] Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.926660 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-credential-socket\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.926756 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-token\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.926791 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-data\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.926822 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-certs\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.926879 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-envoy\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.926912 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqfn\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-kube-api-access-xfqfn\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.926991 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-socket\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.927035 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istiod-ca-cert\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.929049 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.927156 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-podinfo\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.954660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.937448 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-certs\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.967165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.956225 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-credential-socket\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.967165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.956449 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-data\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.967165 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.966659 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istiod-ca-cert\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:55.967664 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:55.967634 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-socket\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:56.009661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.009564 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-envoy\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:56.146702 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.146670 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqfn\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-kube-api-access-xfqfn\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:56.198655 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.169814 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-podinfo\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:56.207722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.199468 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-token\") pod \"e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:56.358091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.358005 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:56.804948 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:54:56.804920 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a43c0dc_f3bc_4b23_a52b_9b548d3767b6.slice/crio-be7f3ceac11011e96f1b492f9d80e3fe45baacc75634f4318d16c0df4ec84e7f WatchSource:0}: Error finding container be7f3ceac11011e96f1b492f9d80e3fe45baacc75634f4318d16c0df4ec84e7f: Status 404 returned error can't find the container with id be7f3ceac11011e96f1b492f9d80e3fe45baacc75634f4318d16c0df4ec84e7f Nov 01 00:54:56.807172 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.807148 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn"] Nov 01 00:54:56.808285 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.808243 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:54:56.808366 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.808327 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:54:56.808415 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:56.808368 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:54:57.774266 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:57.774224 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" event={"ID":"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6","Type":"ContainerStarted","Data":"08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b"} Nov 01 00:54:57.774427 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:57.774288 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" event={"ID":"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6","Type":"ContainerStarted","Data":"be7f3ceac11011e96f1b492f9d80e3fe45baacc75634f4318d16c0df4ec84e7f"} Nov 01 00:54:57.947090 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:57.947035 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podStartSLOduration=2.947022742 podStartE2EDuration="2.947022742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:54:57.943868972 +0000 UTC m=+1887.923451300" watchObservedRunningTime="2026-11-01 00:54:57.947022742 +0000 UTC m=+1887.926605077" Nov 01 00:54:58.358613 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:58.358566 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:58.366364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:58.365098 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:58.783836 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:58.783806 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:54:58.784761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:54:58.784739 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:55:02.056555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:02.056518 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb"] Nov 01 00:55:02.100958 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:02.100924 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526a5487-g6ztb"] Nov 01 00:55:02.195915 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:02.195885 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b04cc1-e01d-48be-a56c-2606e48b6704" path="/var/lib/kubelet/pods/f4b04cc1-e01d-48be-a56c-2606e48b6704/volumes" Nov 01 00:55:07.054198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:07.054142 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd"] Nov 01 00:55:07.099279 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:07.099249 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77141d38-phxrd"] Nov 01 00:55:08.050966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:08.050929 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf"] Nov 01 00:55:08.100966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:08.100929 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe7138-5t5gf"] Nov 01 00:55:08.193727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:08.193688 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e800fb-e72f-4961-8f97-88baad1a6f4d" path="/var/lib/kubelet/pods/c9e800fb-e72f-4961-8f97-88baad1a6f4d/volumes" Nov 01 00:55:08.194258 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:08.194243 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72699e9-14fb-4cd4-90ac-676afb83f9c9" path="/var/lib/kubelet/pods/e72699e9-14fb-4cd4-90ac-676afb83f9c9/volumes" Nov 01 00:55:13.104830 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:55:13.104793 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:55:21.053254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:21.053223 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn"] Nov 01 00:55:21.103077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:21.103049 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b26548-bprpn"] Nov 01 00:55:22.194893 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:22.194863 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99943c0-c763-4e6c-b773-fd27ab1601bc" path="/var/lib/kubelet/pods/c99943c0-c763-4e6c-b773-fd27ab1601bc/volumes" Nov 01 00:55:25.058126 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:25.058091 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7"] Nov 01 00:55:25.108004 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:25.107970 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l"] Nov 01 00:55:25.112947 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:25.112931 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce1fb8a0-8lr6l"] Nov 01 00:55:25.117190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:25.117166 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f1902c-m9lc7"] Nov 01 00:55:26.193163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:26.193133 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0d5a22-2261-4e21-aa95-60ece65bacf0" path="/var/lib/kubelet/pods/3a0d5a22-2261-4e21-aa95-60ece65bacf0/volumes" Nov 01 00:55:26.193688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:26.193672 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc96513-1914-411e-a293-4e8683794d34" path="/var/lib/kubelet/pods/9cc96513-1914-411e-a293-4e8683794d34/volumes" Nov 01 00:55:27.053117 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:27.053088 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk"] Nov 01 00:55:27.102904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:27.102878 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b16fa68-f98rk"] Nov 01 00:55:28.192798 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:28.192768 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48258d78-1ace-43f7-bbab-661e80719f33" path="/var/lib/kubelet/pods/48258d78-1ace-43f7-bbab-661e80719f33/volumes" Nov 01 00:55:31.006064 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:31.006037 4910 scope.go:122] "RemoveContainer" containerID="5db3e12de71970ffc77f96706596a331758af31237283e7545bbad3ede15fdb0" Nov 01 00:55:31.017436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:31.017405 4910 scope.go:122] "RemoveContainer" containerID="8479ca7a671dff3893edcff724667a38bd6c5c6ea2d58df0e2a05c9d5fe8813a" Nov 01 00:55:31.027761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:31.027741 4910 scope.go:122] "RemoveContainer" containerID="6460655a0bbf2f8ae798c7ae972a45c43ac4f799bd67b9ad84e30f4aed472df5" Nov 01 00:55:31.037330 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:31.037310 4910 scope.go:122] "RemoveContainer" containerID="28d27392c3880589a546dce052b525f83592cc5eb4943bb51d6817e014e813af" Nov 01 00:55:31.068389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:31.068363 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj"] Nov 01 00:55:31.115099 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:31.115075 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a459d458-w6rcj"] Nov 01 00:55:32.195213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:32.195177 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0e9f00-2649-486f-9416-3c285d530a6c" path="/var/lib/kubelet/pods/8a0e9f00-2649-486f-9416-3c285d530a6c/volumes" Nov 01 00:55:33.068045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.068006 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc"] Nov 01 00:55:33.123043 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.123012 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b95067-4prjc"] Nov 01 00:55:33.637880 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.637845 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26"] Nov 01 00:55:33.639080 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.639055 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ddff29-fa3e-49e0-b6f4-71c62187e64f" containerName="istio-proxy" Nov 01 00:55:33.645607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.645583 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.711756 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.711722 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26"] Nov 01 00:55:33.753865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.753833 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-99eb0f-openshift-default-dockercfg-mmzfv\"" Nov 01 00:55:33.834115 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834073 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-podinfo\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834229 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-certs\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834287 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-token\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834320 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-envoy\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834362 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5j7s\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-kube-api-access-q5j7s\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834446 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e820b1-53a8-446d-a90a-20b32e9e08e6-istiod-ca-cert\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834504 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-socket\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834540 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-credential-socket\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.834731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.834585 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-data\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938008 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-podinfo\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938149 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-certs\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938198 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-token\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938232 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-envoy\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938269 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5j7s\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-kube-api-access-q5j7s\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938335 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e820b1-53a8-446d-a90a-20b32e9e08e6-istiod-ca-cert\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938379 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-socket\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938414 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-credential-socket\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.938705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.938453 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-data\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.939341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.939036 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-data\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.946646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.940024 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-certs\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.948785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.947145 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-socket\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.948785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.947789 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e820b1-53a8-446d-a90a-20b32e9e08e6-istiod-ca-cert\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.948785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.948022 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-credential-socket\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.959242 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.959216 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-podinfo\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:33.983048 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:33.982984 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-envoy\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:34.010519 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.010486 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5j7s\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-kube-api-access-q5j7s\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:34.013396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.013345 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-token\") pod \"e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:34.202013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.201933 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d" path="/var/lib/kubelet/pods/f5ae5490-5a25-4d8a-b9cb-38e4cdb2f07d/volumes" Nov 01 00:55:34.263823 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.263791 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:34.723746 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:55:34.723699 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e820b1_53a8_446d_a90a_20b32e9e08e6.slice/crio-2e4f7ef1d280985ed8f3276eeb63da8ba4315ab7fe9f285bb2293f76c46bd8c9 WatchSource:0}: Error finding container 2e4f7ef1d280985ed8f3276eeb63da8ba4315ab7fe9f285bb2293f76c46bd8c9: Status 404 returned error can't find the container with id 2e4f7ef1d280985ed8f3276eeb63da8ba4315ab7fe9f285bb2293f76c46bd8c9 Nov 01 00:55:34.725301 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.725274 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26"] Nov 01 00:55:34.726592 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.726555 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:55:34.726719 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.726642 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:55:34.726719 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:34.726675 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:55:35.060283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:35.060247 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc"] Nov 01 00:55:35.067491 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:35.067459 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1daa24-qjqxc"] Nov 01 00:55:35.187162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:35.187127 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" event={"ID":"14e820b1-53a8-446d-a90a-20b32e9e08e6","Type":"ContainerStarted","Data":"799090d7e859b22b90564ea08e1ee7210f6819e79d38ae39fb6a42fab6782dae"} Nov 01 00:55:35.187357 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:35.187181 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" event={"ID":"14e820b1-53a8-446d-a90a-20b32e9e08e6","Type":"ContainerStarted","Data":"2e4f7ef1d280985ed8f3276eeb63da8ba4315ab7fe9f285bb2293f76c46bd8c9"} Nov 01 00:55:35.264919 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:35.264887 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:35.270966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:35.270940 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:35.341903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:35.341804 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podStartSLOduration=2.34178993 podStartE2EDuration="2.34178993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:55:35.338394574 +0000 UTC m=+1925.317976934" watchObservedRunningTime="2026-11-01 00:55:35.34178993 +0000 UTC m=+1925.321372265" Nov 01 00:55:36.197026 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:36.196994 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cf91fa-3388-404e-8d04-4f3352cc1eb0" path="/var/lib/kubelet/pods/98cf91fa-3388-404e-8d04-4f3352cc1eb0/volumes" Nov 01 00:55:36.201240 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:36.201202 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:36.202197 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:36.202176 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:55:43.105129 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:55:43.105035 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:55:44.084432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.084399 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm"] Nov 01 00:55:44.116347 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.116317 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28354599-dsrtm"] Nov 01 00:55:44.198863 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.198825 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f" path="/var/lib/kubelet/pods/d3eb53b1-f81b-4dbc-9270-8a9b8e2d041f/volumes" Nov 01 00:55:44.699954 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.699921 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6"] Nov 01 00:55:44.706100 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.706079 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.761600 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.761569 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6"] Nov 01 00:55:44.790293 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.790256 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.790687 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.790659 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zdp\" (UniqueName: \"kubernetes.io/projected/da0f43c7-6306-40bb-9246-311a938fbfdd-kube-api-access-47zdp\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.790810 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.790728 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.803004 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.802968 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:55:44.803137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.803028 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:55:44.892006 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.891972 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zdp\" (UniqueName: \"kubernetes.io/projected/da0f43c7-6306-40bb-9246-311a938fbfdd-kube-api-access-47zdp\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.892198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.892030 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.892198 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.892074 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.892737 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.892715 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.894785 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.894764 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:44.953731 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:44.953663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zdp\" (UniqueName: \"kubernetes.io/projected/da0f43c7-6306-40bb-9246-311a938fbfdd-kube-api-access-47zdp\") pod \"maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:45.021557 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:45.021532 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:45.404607 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:55:45.404568 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0f43c7_6306_40bb_9246_311a938fbfdd.slice/crio-3ef1be698c300c665af4c380310fd3cdfe7987731220c9f2bfab81020b0aea85 WatchSource:0}: Error finding container 3ef1be698c300c665af4c380310fd3cdfe7987731220c9f2bfab81020b0aea85: Status 404 returned error can't find the container with id 3ef1be698c300c665af4c380310fd3cdfe7987731220c9f2bfab81020b0aea85 Nov 01 00:55:45.407840 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:45.407815 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6"] Nov 01 00:55:46.302957 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:46.302919 4910 generic.go:356] "Generic (PLEG): container finished" podID="da0f43c7-6306-40bb-9246-311a938fbfdd" containerID="ca92f6a6f76c0e724e7842b972bbd338b55536a007578c4538a125bd87867042" exitCode=0 Nov 01 00:55:46.303364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:46.303290 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" event={"ID":"da0f43c7-6306-40bb-9246-311a938fbfdd","Type":"ContainerDied","Data":"ca92f6a6f76c0e724e7842b972bbd338b55536a007578c4538a125bd87867042"} Nov 01 00:55:46.303532 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:46.303434 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" event={"ID":"da0f43c7-6306-40bb-9246-311a938fbfdd","Type":"ContainerStarted","Data":"3ef1be698c300c665af4c380310fd3cdfe7987731220c9f2bfab81020b0aea85"} Nov 01 00:55:47.500527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.500502 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:47.633287 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.633191 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-xks-ca\") pod \"da0f43c7-6306-40bb-9246-311a938fbfdd\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " Nov 01 00:55:47.633462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.633309 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0f43c7-6306-40bb-9246-311a938fbfdd-kube-api-access-47zdp\" (UniqueName: \"kubernetes.io/projected/da0f43c7-6306-40bb-9246-311a938fbfdd-kube-api-access-47zdp\") pod \"da0f43c7-6306-40bb-9246-311a938fbfdd\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " Nov 01 00:55:47.633462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.633407 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-openshift-ca\") pod \"da0f43c7-6306-40bb-9246-311a938fbfdd\" (UID: \"da0f43c7-6306-40bb-9246-311a938fbfdd\") " Nov 01 00:55:47.634101 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.634068 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-openshift-ca" pod "da0f43c7-6306-40bb-9246-311a938fbfdd" (UID: "da0f43c7-6306-40bb-9246-311a938fbfdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:55:47.634325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.634304 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:55:47.636008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.635971 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-xks-ca" pod "da0f43c7-6306-40bb-9246-311a938fbfdd" (UID: "da0f43c7-6306-40bb-9246-311a938fbfdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:55:47.636302 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.636278 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0f43c7-6306-40bb-9246-311a938fbfdd-kube-api-access-47zdp" pod "da0f43c7-6306-40bb-9246-311a938fbfdd" (UID: "da0f43c7-6306-40bb-9246-311a938fbfdd"). InnerVolumeSpecName "kube-api-access-47zdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:55:47.735488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.735450 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zdp\" (UniqueName: \"kubernetes.io/projected/da0f43c7-6306-40bb-9246-311a938fbfdd-kube-api-access-47zdp\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:55:47.735488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:47.735482 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0f43c7-6306-40bb-9246-311a938fbfdd-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:55:48.060325 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.060289 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k"] Nov 01 00:55:48.065462 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.065433 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5d94fa-j5l8k"] Nov 01 00:55:48.193794 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.193755 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f724eb2-134b-43a5-9517-f65caa770eb1" path="/var/lib/kubelet/pods/3f724eb2-134b-43a5-9517-f65caa770eb1/volumes" Nov 01 00:55:48.322324 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.322241 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" Nov 01 00:55:48.322324 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.322271 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6" event={"ID":"da0f43c7-6306-40bb-9246-311a938fbfdd","Type":"ContainerDied","Data":"3ef1be698c300c665af4c380310fd3cdfe7987731220c9f2bfab81020b0aea85"} Nov 01 00:55:48.322324 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.322316 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ef1be698c300c665af4c380310fd3cdfe7987731220c9f2bfab81020b0aea85" Nov 01 00:55:48.491678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.481108 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7"] Nov 01 00:55:48.491678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.486689 4910 cpu_manager.go:414] "containerMap: removing container" podUID="da0f43c7-6306-40bb-9246-311a938fbfdd" containerName="revoke-keys" Nov 01 00:55:48.491678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.486715 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0f43c7-6306-40bb-9246-311a938fbfdd" containerName="revoke-keys" Nov 01 00:55:48.491678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.486916 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0f43c7-6306-40bb-9246-311a938fbfdd" containerName="revoke-keys" Nov 01 00:55:48.496060 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.496034 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.543452 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.543420 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7"] Nov 01 00:55:48.548684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.548126 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt785\" (UniqueName: \"kubernetes.io/projected/6242a453-b711-46f3-aaf4-7682a38c4f91-kube-api-access-rt785\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.548684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.548336 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.548684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.548484 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.552579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.552534 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:55:48.553110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.553050 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:55:48.649457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.649373 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.649645 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.649479 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.649645 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.649545 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt785\" (UniqueName: \"kubernetes.io/projected/6242a453-b711-46f3-aaf4-7682a38c4f91-kube-api-access-rt785\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.650223 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.650203 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.652047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.652021 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.711288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.711262 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt785\" (UniqueName: \"kubernetes.io/projected/6242a453-b711-46f3-aaf4-7682a38c4f91-kube-api-access-rt785\") pod \"maas-api-revoke-keys-e2e-derive-5f10727e-fztk7\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:48.815385 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:48.815350 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:49.049330 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:49.049305 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl"] Nov 01 00:55:49.054007 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:49.053986 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f04d4b-ljqsl"] Nov 01 00:55:49.201532 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:55:49.201502 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6242a453_b711_46f3_aaf4_7682a38c4f91.slice/crio-3bca32aa8b1450d0e044dd481434c7e817a32b917b10b6e640aa97ef600771fb WatchSource:0}: Error finding container 3bca32aa8b1450d0e044dd481434c7e817a32b917b10b6e640aa97ef600771fb: Status 404 returned error can't find the container with id 3bca32aa8b1450d0e044dd481434c7e817a32b917b10b6e640aa97ef600771fb Nov 01 00:55:49.202826 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:49.202805 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7"] Nov 01 00:55:49.333247 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:49.333163 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" event={"ID":"6242a453-b711-46f3-aaf4-7682a38c4f91","Type":"ContainerStarted","Data":"3bca32aa8b1450d0e044dd481434c7e817a32b917b10b6e640aa97ef600771fb"} Nov 01 00:55:50.054141 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:50.054113 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w"] Nov 01 00:55:50.059185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:50.059157 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf54b5e-ckk6w"] Nov 01 00:55:50.196794 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:50.196763 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d021e4-e076-43a9-a60d-3d62a728f24d" path="/var/lib/kubelet/pods/63d021e4-e076-43a9-a60d-3d62a728f24d/volumes" Nov 01 00:55:50.197344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:50.197322 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700bcef9-a81d-4bdb-84ae-492d2b1bb50f" path="/var/lib/kubelet/pods/700bcef9-a81d-4bdb-84ae-492d2b1bb50f/volumes" Nov 01 00:55:50.347496 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:50.347404 4910 generic.go:356] "Generic (PLEG): container finished" podID="6242a453-b711-46f3-aaf4-7682a38c4f91" containerID="751cb2ad1b28d8b0116bb1c5ef9376f05d2f8d11779852f6f580f47bfa53129f" exitCode=0 Nov 01 00:55:50.347677 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:50.347497 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" event={"ID":"6242a453-b711-46f3-aaf4-7682a38c4f91","Type":"ContainerDied","Data":"751cb2ad1b28d8b0116bb1c5ef9376f05d2f8d11779852f6f580f47bfa53129f"} Nov 01 00:55:51.576989 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.576962 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:51.711183 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.711101 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-openshift-ca\") pod \"6242a453-b711-46f3-aaf4-7682a38c4f91\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " Nov 01 00:55:51.711354 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.711235 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6242a453-b711-46f3-aaf4-7682a38c4f91-kube-api-access-rt785\" (UniqueName: \"kubernetes.io/projected/6242a453-b711-46f3-aaf4-7682a38c4f91-kube-api-access-rt785\") pod \"6242a453-b711-46f3-aaf4-7682a38c4f91\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " Nov 01 00:55:51.711431 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.711412 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-xks-ca\") pod \"6242a453-b711-46f3-aaf4-7682a38c4f91\" (UID: \"6242a453-b711-46f3-aaf4-7682a38c4f91\") " Nov 01 00:55:51.711536 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.711509 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-openshift-ca" pod "6242a453-b711-46f3-aaf4-7682a38c4f91" (UID: "6242a453-b711-46f3-aaf4-7682a38c4f91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:55:51.712145 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.712121 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:55:51.714299 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.714273 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-xks-ca" pod "6242a453-b711-46f3-aaf4-7682a38c4f91" (UID: "6242a453-b711-46f3-aaf4-7682a38c4f91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:55:51.714814 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.714782 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6242a453-b711-46f3-aaf4-7682a38c4f91-kube-api-access-rt785" pod "6242a453-b711-46f3-aaf4-7682a38c4f91" (UID: "6242a453-b711-46f3-aaf4-7682a38c4f91"). InnerVolumeSpecName "kube-api-access-rt785". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:55:51.813370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.813340 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6242a453-b711-46f3-aaf4-7682a38c4f91-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:55:51.813370 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:51.813369 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt785\" (UniqueName: \"kubernetes.io/projected/6242a453-b711-46f3-aaf4-7682a38c4f91-kube-api-access-rt785\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:55:52.371498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:52.371472 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" Nov 01 00:55:52.371735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:52.371518 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7" event={"ID":"6242a453-b711-46f3-aaf4-7682a38c4f91","Type":"ContainerDied","Data":"3bca32aa8b1450d0e044dd481434c7e817a32b917b10b6e640aa97ef600771fb"} Nov 01 00:55:52.371735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:55:52.371573 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bca32aa8b1450d0e044dd481434c7e817a32b917b10b6e640aa97ef600771fb" Nov 01 00:56:03.529768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.529157 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4"] Nov 01 00:56:03.532162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.532118 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6242a453-b711-46f3-aaf4-7682a38c4f91" containerName="revoke-keys" Nov 01 00:56:03.532423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.532402 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6242a453-b711-46f3-aaf4-7682a38c4f91" containerName="revoke-keys" Nov 01 00:56:03.538817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.538797 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6242a453-b711-46f3-aaf4-7682a38c4f91" containerName="revoke-keys" Nov 01 00:56:03.559941 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.559913 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.642545 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.642511 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b112c158-openshift-default-dockercfg-jnjjd\"" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773497 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773531 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2w7\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-kube-api-access-ch2w7\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773615 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773683 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773708 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773792 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773870 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.778652 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.773999 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.798964 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.798842 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4"] Nov 01 00:56:03.876217 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.876184 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.876575 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.876549 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.876943 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.876917 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.877077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.877062 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2w7\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-kube-api-access-ch2w7\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.877233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.877218 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.877364 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.877349 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.877511 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.877495 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.877644 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.877612 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.877824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.877805 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.878579 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.878542 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.885013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.881088 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.885013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.881367 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.885013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.881600 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.886066 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.886047 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.892126 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.892074 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:03.899672 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:03.898654 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:04.035654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:04.027517 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:04.058707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:04.055246 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2w7\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-kube-api-access-ch2w7\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:04.247647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:04.247262 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:04.761280 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:04.761245 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4"] Nov 01 00:56:04.795850 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:56:04.795806 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e9c1df_cc91_4aed_8e1a_eb51042c031d.slice/crio-c5da9d938bf33be30e5f3c228c863ea055b72b021132a79ddae7192d8c97e03a WatchSource:0}: Error finding container c5da9d938bf33be30e5f3c228c863ea055b72b021132a79ddae7192d8c97e03a: Status 404 returned error can't find the container with id c5da9d938bf33be30e5f3c228c863ea055b72b021132a79ddae7192d8c97e03a Nov 01 00:56:04.800175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:04.800126 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:04.800274 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:04.800234 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:04.800320 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:04.800298 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:05.543127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:05.543083 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" event={"ID":"c3e9c1df-cc91-4aed-8e1a-eb51042c031d","Type":"ContainerStarted","Data":"34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b"} Nov 01 00:56:05.543438 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:05.543419 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" event={"ID":"c3e9c1df-cc91-4aed-8e1a-eb51042c031d","Type":"ContainerStarted","Data":"c5da9d938bf33be30e5f3c228c863ea055b72b021132a79ddae7192d8c97e03a"} Nov 01 00:56:05.719478 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:05.719422 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podStartSLOduration=2.719405456 podStartE2EDuration="2.719405456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:56:05.665283071 +0000 UTC m=+1955.644865405" watchObservedRunningTime="2026-11-01 00:56:05.719405456 +0000 UTC m=+1955.698987791" Nov 01 00:56:06.248688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:06.248658 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:06.254083 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:06.254060 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:06.553750 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:06.553659 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:06.554687 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:06.554664 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:56:07.050680 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:07.050605 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh"] Nov 01 00:56:07.055383 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:07.055362 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c89ea6-524mh"] Nov 01 00:56:08.054499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:08.054448 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh"] Nov 01 00:56:08.113237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:08.113200 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l"] Nov 01 00:56:08.118172 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:08.118147 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf9ebf8-vxzvh"] Nov 01 00:56:08.123216 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:08.123190 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72be219b-p528l"] Nov 01 00:56:08.196346 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:08.196317 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34425d1e-8f64-4332-b32f-85a88b8bb881" path="/var/lib/kubelet/pods/34425d1e-8f64-4332-b32f-85a88b8bb881/volumes" Nov 01 00:56:08.196862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:08.196846 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529bb719-82c9-48e9-9845-dfe955e965c2" path="/var/lib/kubelet/pods/529bb719-82c9-48e9-9845-dfe955e965c2/volumes" Nov 01 00:56:08.197302 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:08.197289 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5041663-87fc-4806-83d0-185c303afe2b" path="/var/lib/kubelet/pods/c5041663-87fc-4806-83d0-185c303afe2b/volumes" Nov 01 00:56:13.107036 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:56:13.107001 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:56:14.063213 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:14.063178 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2"] Nov 01 00:56:14.108900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:14.108868 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b68245-82wr2"] Nov 01 00:56:14.197755 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:14.197711 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8804ef-0225-471a-8429-43442211dcdb" path="/var/lib/kubelet/pods/6d8804ef-0225-471a-8429-43442211dcdb/volumes" Nov 01 00:56:20.535183 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.535132 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh"] Nov 01 00:56:20.545464 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.545435 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.597397 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.597362 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh"] Nov 01 00:56:20.641153 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.641116 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:56:20.641327 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.641220 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:56:20.693929 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.693893 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2qr\" (UniqueName: \"kubernetes.io/projected/fb56c835-1d5e-48fa-944e-4adee75df049-kube-api-access-vl2qr\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.694138 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.694118 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.694238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.694223 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.796192 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.796103 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2qr\" (UniqueName: \"kubernetes.io/projected/fb56c835-1d5e-48fa-944e-4adee75df049-kube-api-access-vl2qr\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.796378 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.796265 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.796378 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.796320 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.796979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.796960 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.798934 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.798913 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:20.865271 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:20.865237 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2qr\" (UniqueName: \"kubernetes.io/projected/fb56c835-1d5e-48fa-944e-4adee75df049-kube-api-access-vl2qr\") pod \"maas-api-revoke-sub-cd71bc49-bxnbh\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:21.165002 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:21.164914 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:21.581973 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:56:21.581924 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb56c835_1d5e_48fa_944e_4adee75df049.slice/crio-74cba18beb1ab3975a0eafbbab3e4757be085fdfaec258852814f64d0d819103 WatchSource:0}: Error finding container 74cba18beb1ab3975a0eafbbab3e4757be085fdfaec258852814f64d0d819103: Status 404 returned error can't find the container with id 74cba18beb1ab3975a0eafbbab3e4757be085fdfaec258852814f64d0d819103 Nov 01 00:56:21.585798 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:21.585764 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh"] Nov 01 00:56:21.706576 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:21.706543 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" event={"ID":"fb56c835-1d5e-48fa-944e-4adee75df049","Type":"ContainerStarted","Data":"74cba18beb1ab3975a0eafbbab3e4757be085fdfaec258852814f64d0d819103"} Nov 01 00:56:22.724272 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:22.724236 4910 generic.go:356] "Generic (PLEG): container finished" podID="fb56c835-1d5e-48fa-944e-4adee75df049" containerID="ae040e71482d4cabd1b27a60e64e1e7853551b5cf4c6cb9610aa6ea809fcc297" exitCode=0 Nov 01 00:56:22.724613 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:22.724321 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" event={"ID":"fb56c835-1d5e-48fa-944e-4adee75df049","Type":"ContainerDied","Data":"ae040e71482d4cabd1b27a60e64e1e7853551b5cf4c6cb9610aa6ea809fcc297"} Nov 01 00:56:24.047077 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.047042 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:24.091653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.091579 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-xks-ca\") pod \"fb56c835-1d5e-48fa-944e-4adee75df049\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " Nov 01 00:56:24.091874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.091750 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb56c835-1d5e-48fa-944e-4adee75df049-kube-api-access-vl2qr\" (UniqueName: \"kubernetes.io/projected/fb56c835-1d5e-48fa-944e-4adee75df049-kube-api-access-vl2qr\") pod \"fb56c835-1d5e-48fa-944e-4adee75df049\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " Nov 01 00:56:24.091945 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.091881 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-openshift-ca\") pod \"fb56c835-1d5e-48fa-944e-4adee75df049\" (UID: \"fb56c835-1d5e-48fa-944e-4adee75df049\") " Nov 01 00:56:24.092510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.092457 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-openshift-ca" pod "fb56c835-1d5e-48fa-944e-4adee75df049" (UID: "fb56c835-1d5e-48fa-944e-4adee75df049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:24.097062 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.097040 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-xks-ca" pod "fb56c835-1d5e-48fa-944e-4adee75df049" (UID: "fb56c835-1d5e-48fa-944e-4adee75df049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:56:24.097251 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.097234 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb56c835-1d5e-48fa-944e-4adee75df049-kube-api-access-vl2qr" pod "fb56c835-1d5e-48fa-944e-4adee75df049" (UID: "fb56c835-1d5e-48fa-944e-4adee75df049"). InnerVolumeSpecName "kube-api-access-vl2qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:24.193722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.193680 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2qr\" (UniqueName: \"kubernetes.io/projected/fb56c835-1d5e-48fa-944e-4adee75df049-kube-api-access-vl2qr\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:24.193722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.193725 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:24.194137 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.193741 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb56c835-1d5e-48fa-944e-4adee75df049-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:24.745611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.745579 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" Nov 01 00:56:24.745611 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.745600 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh" event={"ID":"fb56c835-1d5e-48fa-944e-4adee75df049","Type":"ContainerDied","Data":"74cba18beb1ab3975a0eafbbab3e4757be085fdfaec258852814f64d0d819103"} Nov 01 00:56:24.745857 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:24.745660 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74cba18beb1ab3975a0eafbbab3e4757be085fdfaec258852814f64d0d819103" Nov 01 00:56:25.048773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:25.048742 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r"] Nov 01 00:56:25.076824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:25.076790 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08827e-nf95r"] Nov 01 00:56:25.546690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:25.546651 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5"] Nov 01 00:56:25.547257 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:25.547192 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" containerID="cri-o://4848bab028847c1cd5e9a55a1cdf68f1754abd2d3565a24353f906d940b0e13e" gracePeriod=30 Nov 01 00:56:26.194499 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:26.194468 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46e6670-e8ea-487b-b760-3a6d5ef34174" path="/var/lib/kubelet/pods/a46e6670-e8ea-487b-b760-3a6d5ef34174/volumes" Nov 01 00:56:29.825849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:29.825791 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:56:29.826303 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:29.825902 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:56:30.364426 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.364377 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:56:30.364843 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.364807 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:56:30.863118 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.863076 4910 generic.go:356] "Generic (PLEG): container finished" podID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerID="4848bab028847c1cd5e9a55a1cdf68f1754abd2d3565a24353f906d940b0e13e" exitCode=0 Nov 01 00:56:30.863555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.863151 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" event={"ID":"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce","Type":"ContainerDied","Data":"4848bab028847c1cd5e9a55a1cdf68f1754abd2d3565a24353f906d940b0e13e"} Nov 01 00:56:30.863555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.863244 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" event={"ID":"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce","Type":"ContainerDied","Data":"302e83309be1022b190f28ff70a438734d16d5a163fc84b145cdbf6cef17d8cc"} Nov 01 00:56:30.863555 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.863289 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="302e83309be1022b190f28ff70a438734d16d5a163fc84b145cdbf6cef17d8cc" Nov 01 00:56:30.864037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.864018 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:56:30.915902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.915829 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-envoy\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916068 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916048 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-credential-socket\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916195 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-certs\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916272 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916228 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istiod-ca-cert\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916272 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916259 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-data\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916286 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-token\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-token\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916349 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-credential-socket" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:30.916501 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916390 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-podinfo\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916501 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916426 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-kube-api-access-mz2pn\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-kube-api-access-mz2pn\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916441 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-certs" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:30.916837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916465 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-socket\") pod \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\" (UID: \"098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce\") " Nov 01 00:56:30.916837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.916569 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-data" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:30.917214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.917191 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-socket" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:30.917331 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.917301 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istiod-ca-cert" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:56:30.917417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.917398 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:30.917477 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.917424 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:30.917477 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.917440 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:30.917477 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.917453 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:30.918670 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.918650 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-token" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:30.918895 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.918868 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-envoy" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:56:30.919396 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.919375 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-podinfo" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:56:30.919848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:30.919829 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-kube-api-access-mz2pn" pod "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" (UID: "098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce"). InnerVolumeSpecName "kube-api-access-mz2pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:56:31.019292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.019262 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.019292 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.019291 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.019490 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.019302 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.019490 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.019312 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz2pn\" (UniqueName: \"kubernetes.io/projected/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-kube-api-access-mz2pn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.019490 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.019322 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:56:31.102326 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.102289 4910 scope.go:122] "RemoveContainer" containerID="45a6c293a4c3189339a50f183c1b41320bdad1eb1449f4f580b97789e1dcbd83" Nov 01 00:56:31.119128 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.119100 4910 scope.go:122] "RemoveContainer" containerID="4848bab028847c1cd5e9a55a1cdf68f1754abd2d3565a24353f906d940b0e13e" Nov 01 00:56:31.135469 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.135443 4910 scope.go:122] "RemoveContainer" containerID="178412defffbe2bdab24443fe8631ff5e9debbce45fbf2cfb3608e974e2f38a5" Nov 01 00:56:31.145983 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.145964 4910 scope.go:122] "RemoveContainer" containerID="c97424f858faa00424a5a1a244117d715033d96b1a22b40f922b1fc13401a58d" Nov 01 00:56:31.156295 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.156271 4910 scope.go:122] "RemoveContainer" containerID="72de6f4e20ec30a1ec28ee444f45e71cb8695efc433492c1a02187670f026f25" Nov 01 00:56:31.168934 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.168911 4910 scope.go:122] "RemoveContainer" containerID="7536fbbee52df3786e2edea667908f4d8be604f9bb5ad8a5754c04dbd2189aa8" Nov 01 00:56:31.181980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.181955 4910 scope.go:122] "RemoveContainer" containerID="ecae26c05777c20f424a3fdcf98b76371ebe3b209ef96f4be6a2e4764443d39e" Nov 01 00:56:31.194876 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.194854 4910 scope.go:122] "RemoveContainer" containerID="5e761aa86022e8ed0d0b6514209bbb249b205faf12a2edc3f87f4b2ad6c041e3" Nov 01 00:56:31.223806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.220131 4910 scope.go:122] "RemoveContainer" containerID="0f5d00dc77d3970f12c9a1d1c08fe525807d2952b99998190210bf05cbcab3a1" Nov 01 00:56:31.267110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.266388 4910 scope.go:122] "RemoveContainer" containerID="4aca68c283009c9ef1306512f8c186f1507c87641a646ca2760ea7865ec30d11" Nov 01 00:56:31.300253 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.300050 4910 scope.go:122] "RemoveContainer" containerID="32102761f02daaabec9f500fa5071a19ab6cd42d68af997fa5edda2eda7afc33" Nov 01 00:56:31.332027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.331994 4910 scope.go:122] "RemoveContainer" containerID="3e648724823f5140a86d23debfc956adcd47299a96870b95f66e896b437615f1" Nov 01 00:56:31.353092 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.353067 4910 scope.go:122] "RemoveContainer" containerID="dd45525d28d06ba2db3d4da4b5c419d58ec2e867a9acfff392aba5a1000a6c0f" Nov 01 00:56:31.379482 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.379458 4910 scope.go:122] "RemoveContainer" containerID="72efdfda95df7585ea107baec4ad47ad1a8f28116c72494cdcee91363e7dc3c3" Nov 01 00:56:31.404263 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.404208 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6"] Nov 01 00:56:31.416914 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.411806 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13f6ab63-rcdk6"] Nov 01 00:56:31.877023 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.876987 4910 generic.go:356] "Generic (PLEG): container finished" podID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerID="08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b" exitCode=0 Nov 01 00:56:31.877538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.877063 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" event={"ID":"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6","Type":"ContainerDied","Data":"08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b"} Nov 01 00:56:31.877538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.877148 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5" Nov 01 00:56:31.878075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.878057 4910 scope.go:122] "RemoveContainer" containerID="08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b" Nov 01 00:56:31.885248 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.885212 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:31.885407 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.885301 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:31.885407 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:31.885345 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:56:32.206529 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:32.206450 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0f43c7-6306-40bb-9246-311a938fbfdd" path="/var/lib/kubelet/pods/da0f43c7-6306-40bb-9246-311a938fbfdd/volumes" Nov 01 00:56:32.284424 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:32.284376 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5"] Nov 01 00:56:32.294102 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:32.294035 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-w8jl5"] Nov 01 00:56:32.898405 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:32.898356 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" event={"ID":"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6","Type":"ContainerStarted","Data":"a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a"} Nov 01 00:56:33.369121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:33.368058 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:56:33.369121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:33.368197 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 01 00:56:33.369121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:33.368245 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 01 00:56:34.200200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:34.199955 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" path="/var/lib/kubelet/pods/098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce/volumes" Nov 01 00:56:34.358990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:34.358954 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 01 00:56:34.359310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:34.359261 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 01 00:56:35.365649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:35.364599 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 01 00:56:35.365649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:35.364679 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 01 00:56:36.075398 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:36.075367 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz"] Nov 01 00:56:36.099288 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:36.099261 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b3da1b-w7wnz"] Nov 01 00:56:36.199290 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:36.199255 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d3cff4-6863-454c-ad90-7e9a93d291b9" path="/var/lib/kubelet/pods/63d3cff4-6863-454c-ad90-7e9a93d291b9/volumes" Nov 01 00:56:36.359703 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:36.359582 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 01 00:56:36.359967 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:36.359936 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 01 00:56:37.360756 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:37.360719 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 01 00:56:37.361340 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:37.361307 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 01 00:56:37.798787 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:37.798753 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7"] Nov 01 00:56:37.803436 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:37.803411 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5f10727e-fztk7"] Nov 01 00:56:38.193852 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:38.193777 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6242a453-b711-46f3-aaf4-7682a38c4f91" path="/var/lib/kubelet/pods/6242a453-b711-46f3-aaf4-7682a38c4f91/volumes" Nov 01 00:56:38.358985 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:38.358947 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 01 00:56:38.359271 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:38.359241 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 01 00:56:39.360095 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:39.360062 4910 patch_prober.go:29] interesting pod/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 01 00:56:39.361216 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:39.361177 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 01 00:56:40.363601 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:40.363572 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:56:40.364162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:40.364144 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:56:40.364998 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:56:40.364981 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:56:43.232509 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:56:43.232455 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:57:01.881648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:01.881154 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 00:57:02.425803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:02.425758 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:57:02.443612 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:02.443570 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:57:02.544812 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:02.544767 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:57:02.545366 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:02.545336 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 01 00:57:02.721708 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:02.615731 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 01 00:57:02.812880 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:02.812257 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 00:57:03.009709 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:03.003734 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 01 00:57:05.053609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:05.053572 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg"] Nov 01 00:57:05.103660 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:05.103614 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af79cc-k6rgg"] Nov 01 00:57:06.199480 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:06.199443 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa9d9984-6e2e-4316-910e-82da61e64201" path="/var/lib/kubelet/pods/fa9d9984-6e2e-4316-910e-82da61e64201/volumes" Nov 01 00:57:07.247650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.246740 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 00:57:07.247650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.246825 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 00:57:07.271987 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.271922 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn"] Nov 01 00:57:07.299646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.291646 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fb56c835-1d5e-48fa-944e-4adee75df049" containerName="revoke-keys" Nov 01 00:57:07.299646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.291683 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb56c835-1d5e-48fa-944e-4adee75df049" containerName="revoke-keys" Nov 01 00:57:07.299646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.291760 4910 cpu_manager.go:414] "containerMap: removing container" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" Nov 01 00:57:07.299646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.291769 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" Nov 01 00:57:07.299646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.291954 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb56c835-1d5e-48fa-944e-4adee75df049" containerName="revoke-keys" Nov 01 00:57:07.299646 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.291974 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="098fef6a-5bd4-4bda-a5fe-d8d5af5e52ce" containerName="istio-proxy" Nov 01 00:57:07.305645 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.302479 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.377160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.377120 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn"] Nov 01 00:57:07.437865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.437833 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:57:07.440122 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.440098 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:57:07.480694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.479949 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.480694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.480055 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87mj\" (UniqueName: \"kubernetes.io/projected/f807af07-0df2-4d77-bd2d-8c0b5f10413b-kube-api-access-x87mj\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.480694 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.480291 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.582723 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.582325 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.582723 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.582531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.582723 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.582596 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87mj\" (UniqueName: \"kubernetes.io/projected/f807af07-0df2-4d77-bd2d-8c0b5f10413b-kube-api-access-x87mj\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.583558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.583270 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.596398 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.596369 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.644728 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.644702 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87mj\" (UniqueName: \"kubernetes.io/projected/f807af07-0df2-4d77-bd2d-8c0b5f10413b-kube-api-access-x87mj\") pod \"maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:07.934062 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:07.933933 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:08.341343 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:57:08.341309 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf807af07_0df2_4d77_bd2d_8c0b5f10413b.slice/crio-01e7d8ca0f806ad8b9db1b9a9c5b69fb160a8f2bdbd6d8a2b4fb4776055827b6 WatchSource:0}: Error finding container 01e7d8ca0f806ad8b9db1b9a9c5b69fb160a8f2bdbd6d8a2b4fb4776055827b6: Status 404 returned error can't find the container with id 01e7d8ca0f806ad8b9db1b9a9c5b69fb160a8f2bdbd6d8a2b4fb4776055827b6 Nov 01 00:57:08.342027 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.341923 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn"] Nov 01 00:57:08.872525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.872487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" event={"ID":"f807af07-0df2-4d77-bd2d-8c0b5f10413b","Type":"ContainerStarted","Data":"4e6a4635c283b3454d94799ace9fa1df3e41d7eaef47f12235cb431c2513a2f1"} Nov 01 00:57:08.872772 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.872570 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" event={"ID":"f807af07-0df2-4d77-bd2d-8c0b5f10413b","Type":"ContainerStarted","Data":"01e7d8ca0f806ad8b9db1b9a9c5b69fb160a8f2bdbd6d8a2b4fb4776055827b6"} Nov 01 00:57:08.874280 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.874250 4910 generic.go:356] "Generic (PLEG): container finished" podID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerID="799090d7e859b22b90564ea08e1ee7210f6819e79d38ae39fb6a42fab6782dae" exitCode=0 Nov 01 00:57:08.874404 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.874303 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" event={"ID":"14e820b1-53a8-446d-a90a-20b32e9e08e6","Type":"ContainerDied","Data":"799090d7e859b22b90564ea08e1ee7210f6819e79d38ae39fb6a42fab6782dae"} Nov 01 00:57:08.874773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.874733 4910 scope.go:122] "RemoveContainer" containerID="799090d7e859b22b90564ea08e1ee7210f6819e79d38ae39fb6a42fab6782dae" Nov 01 00:57:08.876527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.876492 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:08.876616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.876566 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:08.876616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.876601 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:08.991273 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:08.991220 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" podStartSLOduration=1.9912058689999999 podStartE2EDuration="1.991205869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:57:08.98786735 +0000 UTC m=+2018.967449686" watchObservedRunningTime="2026-11-01 00:57:08.991205869 +0000 UTC m=+2018.970788251" Nov 01 00:57:09.896350 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:09.896318 4910 generic.go:356] "Generic (PLEG): container finished" podID="f807af07-0df2-4d77-bd2d-8c0b5f10413b" containerID="4e6a4635c283b3454d94799ace9fa1df3e41d7eaef47f12235cb431c2513a2f1" exitCode=0 Nov 01 00:57:09.896899 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:09.896407 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" event={"ID":"f807af07-0df2-4d77-bd2d-8c0b5f10413b","Type":"ContainerDied","Data":"4e6a4635c283b3454d94799ace9fa1df3e41d7eaef47f12235cb431c2513a2f1"} Nov 01 00:57:09.898918 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:09.898890 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" event={"ID":"14e820b1-53a8-446d-a90a-20b32e9e08e6","Type":"ContainerStarted","Data":"61ec14e801f61940fbda1d98074d0c97df8dc64e42d87e42cdff4bee8475a3a8"} Nov 01 00:57:10.145683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:10.145650 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs"] Nov 01 00:57:10.145931 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:10.145904 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" containerID="cri-o://94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b" gracePeriod=30 Nov 01 00:57:10.302648 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:10.265911 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:57:10.303879 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:10.303847 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 00:57:10.304117 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:10.304087 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 00:57:10.495201 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:10.494239 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x"] Nov 01 00:57:10.495201 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:10.494517 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" containerID="cri-o://12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672" gracePeriod=30 Nov 01 00:57:11.135855 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.135831 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:11.197437 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.197353 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-xks-ca\") pod \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " Nov 01 00:57:11.197572 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.197454 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f807af07-0df2-4d77-bd2d-8c0b5f10413b-kube-api-access-x87mj\" (UniqueName: \"kubernetes.io/projected/f807af07-0df2-4d77-bd2d-8c0b5f10413b-kube-api-access-x87mj\") pod \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " Nov 01 00:57:11.197758 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.197583 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-openshift-ca\") pod \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\" (UID: \"f807af07-0df2-4d77-bd2d-8c0b5f10413b\") " Nov 01 00:57:11.198036 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.198007 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-openshift-ca" pod "f807af07-0df2-4d77-bd2d-8c0b5f10413b" (UID: "f807af07-0df2-4d77-bd2d-8c0b5f10413b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:11.198375 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.198355 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:11.199764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.199741 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-xks-ca" pod "f807af07-0df2-4d77-bd2d-8c0b5f10413b" (UID: "f807af07-0df2-4d77-bd2d-8c0b5f10413b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:57:11.199873 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.199802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f807af07-0df2-4d77-bd2d-8c0b5f10413b-kube-api-access-x87mj" pod "f807af07-0df2-4d77-bd2d-8c0b5f10413b" (UID: "f807af07-0df2-4d77-bd2d-8c0b5f10413b"). InnerVolumeSpecName "kube-api-access-x87mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:11.271657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.271013 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 00:57:11.271657 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.271074 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 00:57:11.300207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.300162 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f807af07-0df2-4d77-bd2d-8c0b5f10413b-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:11.300207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.300207 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87mj\" (UniqueName: \"kubernetes.io/projected/f807af07-0df2-4d77-bd2d-8c0b5f10413b-kube-api-access-x87mj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:11.921730 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.921692 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" Nov 01 00:57:11.921993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.921733 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn" event={"ID":"f807af07-0df2-4d77-bd2d-8c0b5f10413b","Type":"ContainerDied","Data":"01e7d8ca0f806ad8b9db1b9a9c5b69fb160a8f2bdbd6d8a2b4fb4776055827b6"} Nov 01 00:57:11.921993 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:11.921783 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e7d8ca0f806ad8b9db1b9a9c5b69fb160a8f2bdbd6d8a2b4fb4776055827b6" Nov 01 00:57:12.274653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:12.273029 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 00:57:12.274653 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:12.273096 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 00:57:13.105019 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:13.104985 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:57:13.279681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:13.277279 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 00:57:13.279681 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:13.277359 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 00:57:14.273647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:14.268034 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 00:57:14.273647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:14.268097 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 00:57:15.063445 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.060844 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5"] Nov 01 00:57:15.100976 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.100937 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ec1452-tpjh5"] Nov 01 00:57:15.276487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.276462 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:57:15.278019 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.277913 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:57:15.280172 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.280149 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 00:57:15.326306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.326226 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:57:15.326306 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.326294 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:57:15.446437 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.446414 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:57:15.586607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586518 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-envoy\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.586607 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586575 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309de2b1-bd96-45e9-9e5c-5649dede132d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309de2b1-bd96-45e9-9e5c-5649dede132d-istiod-ca-cert\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.586888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586673 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-podinfo\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.586888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586711 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-kube-api-access-2rhh4\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-kube-api-access-2rhh4\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.586888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586743 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-token\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-token\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.586888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586831 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-data\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.586888 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586884 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-credential-socket\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.587162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.586992 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-certs\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.587162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.587061 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309de2b1-bd96-45e9-9e5c-5649dede132d-istiod-ca-cert" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:15.587162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.587146 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-socket\") pod \"309de2b1-bd96-45e9-9e5c-5649dede132d\" (UID: \"309de2b1-bd96-45e9-9e5c-5649dede132d\") " Nov 01 00:57:15.587465 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.587439 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-credential-socket" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.587616 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.587584 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-certs" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.587730 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.587617 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-socket" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.587730 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.587706 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-data" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.588083 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.588060 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.588205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.588088 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.588205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.588103 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309de2b1-bd96-45e9-9e5c-5649dede132d-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.588205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.588118 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.588205 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.588132 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.589182 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.589160 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-envoy" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.589426 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.589409 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-podinfo" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:15.590054 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.590036 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-kube-api-access-2rhh4" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "kube-api-access-2rhh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:15.590146 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.590072 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-token" pod "309de2b1-bd96-45e9-9e5c-5649dede132d" (UID: "309de2b1-bd96-45e9-9e5c-5649dede132d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:15.690235 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.690198 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.690235 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.690231 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.690417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.690244 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rhh4\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-kube-api-access-2rhh4\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.690417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.690254 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309de2b1-bd96-45e9-9e5c-5649dede132d-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.769047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.769022 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:57:15.894056 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.893971 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-socket\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894056 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894046 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-credential-socket\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894160 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-kube-api-access-8hxgk\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-kube-api-access-8hxgk\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894244 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-certs\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894283 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894279 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-istio-token\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-istio-token\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894324 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-data\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894365 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2972c8-7d7d-4eee-9032-54210747636a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2972c8-7d7d-4eee-9032-54210747636a-istiod-ca-cert\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894441 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894415 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-envoy\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894654 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894446 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd2972c8-7d7d-4eee-9032-54210747636a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2972c8-7d7d-4eee-9032-54210747636a-istio-podinfo\") pod \"bd2972c8-7d7d-4eee-9032-54210747636a\" (UID: \"bd2972c8-7d7d-4eee-9032-54210747636a\") " Nov 01 00:57:15.894969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894944 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-credential-socket" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.894969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.894962 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-certs" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.895160 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.895140 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-socket" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.895238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.895170 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.895238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.895191 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.895464 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.895432 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2972c8-7d7d-4eee-9032-54210747636a-istiod-ca-cert" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:15.897900 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.896419 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-data" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.899297 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.899271 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd2972c8-7d7d-4eee-9032-54210747636a-istio-podinfo" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:15.899743 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.899711 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-envoy" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:15.899835 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.899793 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-kube-api-access-8hxgk" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "kube-api-access-8hxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:15.902081 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.902053 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-istio-token" pod "bd2972c8-7d7d-4eee-9032-54210747636a" (UID: "bd2972c8-7d7d-4eee-9032-54210747636a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:15.990351 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.990314 4910 generic.go:356] "Generic (PLEG): container finished" podID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerID="94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b" exitCode=0 Nov 01 00:57:15.990547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.990365 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" Nov 01 00:57:15.990547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.990404 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" event={"ID":"309de2b1-bd96-45e9-9e5c-5649dede132d","Type":"ContainerDied","Data":"94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b"} Nov 01 00:57:15.990547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.990475 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs" event={"ID":"309de2b1-bd96-45e9-9e5c-5649dede132d","Type":"ContainerDied","Data":"46b0a880ab42f7c04960faafdccb3025f19c23b23ca017356955696ea473c7d6"} Nov 01 00:57:15.990547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.990507 4910 scope.go:122] "RemoveContainer" containerID="94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b" Nov 01 00:57:15.992112 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.992087 4910 generic.go:356] "Generic (PLEG): container finished" podID="bd2972c8-7d7d-4eee-9032-54210747636a" containerID="12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672" exitCode=0 Nov 01 00:57:15.992221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.992150 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" Nov 01 00:57:15.992221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.992162 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" event={"ID":"bd2972c8-7d7d-4eee-9032-54210747636a","Type":"ContainerDied","Data":"12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672"} Nov 01 00:57:15.992221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.992216 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x" event={"ID":"bd2972c8-7d7d-4eee-9032-54210747636a","Type":"ContainerDied","Data":"3cd9b868d9b00999f72f030fdb282a7470143adefd0312ba756a1c22a54e9c0e"} Nov 01 00:57:15.996127 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.996108 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.996244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.996129 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2972c8-7d7d-4eee-9032-54210747636a-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.996244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.996140 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.996244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.996149 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hxgk\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-kube-api-access-8hxgk\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.996244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.996158 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2972c8-7d7d-4eee-9032-54210747636a-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.996244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.996167 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2972c8-7d7d-4eee-9032-54210747636a-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:15.996244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:15.996174 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2972c8-7d7d-4eee-9032-54210747636a-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:16.004221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.004204 4910 scope.go:122] "RemoveContainer" containerID="94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b" Nov 01 00:57:16.004479 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:16.004462 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b\": container with ID starting with 94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b not found: ID does not exist" containerID="94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b" Nov 01 00:57:16.004549 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.004487 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b"} err="failed to get container status \"94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b\": rpc error: code = NotFound desc = could not find container \"94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b\": container with ID starting with 94c6e2d5f03ec3b7ba09569b93f3b61fa6044c7ab3923b66d597e92c209d078b not found: ID does not exist" Nov 01 00:57:16.004549 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.004506 4910 scope.go:122] "RemoveContainer" containerID="12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672" Nov 01 00:57:16.015498 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.015473 4910 scope.go:122] "RemoveContainer" containerID="12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672" Nov 01 00:57:16.015932 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:16.015908 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672\": container with ID starting with 12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672 not found: ID does not exist" containerID="12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672" Nov 01 00:57:16.016015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.015942 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672"} err="failed to get container status \"12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672\": rpc error: code = NotFound desc = could not find container \"12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672\": container with ID starting with 12d20bc7f666a367f4e6c2986ab874e6f4e5f3c734605a217e7c98e1e166f672 not found: ID does not exist" Nov 01 00:57:16.160207 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.160127 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs"] Nov 01 00:57:16.166308 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.166271 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-k8nxs"] Nov 01 00:57:16.193904 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.193874 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b15d641-1837-462a-8743-661c7871f95b" path="/var/lib/kubelet/pods/1b15d641-1837-462a-8743-661c7871f95b/volumes" Nov 01 00:57:16.194398 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.194384 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" path="/var/lib/kubelet/pods/309de2b1-bd96-45e9-9e5c-5649dede132d/volumes" Nov 01 00:57:16.278162 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.278129 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x"] Nov 01 00:57:16.282389 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:16.282369 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-mjl4x"] Nov 01 00:57:18.193148 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:18.193116 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" path="/var/lib/kubelet/pods/bd2972c8-7d7d-4eee-9032-54210747636a/volumes" Nov 01 00:57:21.287034 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.287000 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg"] Nov 01 00:57:21.287547 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.287218 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" containerID="cri-o://0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05" gracePeriod=30 Nov 01 00:57:21.346464 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.346421 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72"] Nov 01 00:57:21.346848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.346796 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" containerID="cri-o://57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400" gracePeriod=30 Nov 01 00:57:21.348313 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.348277 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26"] Nov 01 00:57:21.348703 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.348653 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" containerID="cri-o://58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f" gracePeriod=30 Nov 01 00:57:21.387249 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.387219 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz"] Nov 01 00:57:21.387458 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:21.387424 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" containerID="cri-o://676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2" gracePeriod=30 Nov 01 00:57:25.121280 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:25.121241 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:57:25.121722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:25.121315 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:57:25.173722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:25.173677 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:57:25.173903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:25.173744 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:57:26.684469 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.684155 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:57:26.694739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.694714 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:57:26.761214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.761186 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:57:26.768002 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.767980 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:57:26.774829 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.774805 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-token\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-token\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.774990 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.774975 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-data\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.775113 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775100 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istiod-ca-cert\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.775234 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775219 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-envoy\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.775353 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775339 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-podinfo\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.775457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775446 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istiod-ca-cert\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.775574 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775561 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-certs\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.775735 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775721 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-credential-socket\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.775865 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775853 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-certs\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.775955 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.775459 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-data" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.776119 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776100 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-kube-api-access-8j4wb\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-kube-api-access-8j4wb\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.776245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776232 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-credential-socket\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.776366 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776354 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-socket\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.776518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776505 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-token\") pod \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\" (UID: \"2d5a8676-5f71-4c35-aed8-b373c47c3a9b\") " Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776737 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-socket\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776832 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-kube-api-access-fxv6z\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-kube-api-access-fxv6z\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776890 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-podinfo\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776916 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-envoy\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776980 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-data\") pod \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\" (UID: \"15803f9b-8f6f-4169-8e00-4b62fcc1e09d\") " Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776517 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-certs" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776851 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istiod-ca-cert" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.776945 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-credential-socket" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.777174 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-socket" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.777852 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-certs" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.777876 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-socket" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.778124 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istiod-ca-cert" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.778183 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-credential-socket" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.778564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.778526 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-token" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.778932 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.778957 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.778972 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.778987 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.779003 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.779018 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.779031 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.779045 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.779060 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.779075 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.779453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.779136 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-data" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.782245 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.782211 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-envoy" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.783047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.783013 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-kube-api-access-fxv6z" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "kube-api-access-fxv6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.784386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.784347 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-envoy" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.784386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.784357 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-podinfo" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:26.784493 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.784390 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-kube-api-access-8j4wb" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "kube-api-access-8j4wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.784638 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.784605 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-podinfo" pod "15803f9b-8f6f-4169-8e00-4b62fcc1e09d" (UID: "15803f9b-8f6f-4169-8e00-4b62fcc1e09d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:26.786091 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.786072 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-token" pod "2d5a8676-5f71-4c35-aed8-b373c47c3a9b" (UID: "2d5a8676-5f71-4c35-aed8-b373c47c3a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.879803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.879766 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-certs\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.879824 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-kube-api-access-2944m\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.879858 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-istio-token\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-istio-token\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.879920 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-certs\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880013 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.879982 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-socket\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880043 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-credential-socket\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880077 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-kube-api-access-fzm6q\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-kube-api-access-fzm6q\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880105 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-data\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880131 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-token\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-token\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880207 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a05687ed-54f9-4637-a3d6-451b174895e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05687ed-54f9-4637-a3d6-451b174895e4-istio-podinfo\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880255 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880235 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530d5660-94c7-4ae6-a7c4-e825d375d84d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530d5660-94c7-4ae6-a7c4-e825d375d84d-istiod-ca-cert\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880386 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-socket\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880473 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-envoy\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880553 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880523 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-data\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880561 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-podinfo\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880588 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-envoy\") pod \"530d5660-94c7-4ae6-a7c4-e825d375d84d\" (UID: \"530d5660-94c7-4ae6-a7c4-e825d375d84d\") " Nov 01 00:57:26.880782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880706 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05687ed-54f9-4637-a3d6-451b174895e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05687ed-54f9-4637-a3d6-451b174895e4-istiod-ca-cert\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880737 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-credential-socket\") pod \"a05687ed-54f9-4637-a3d6-451b174895e4\" (UID: \"a05687ed-54f9-4637-a3d6-451b174895e4\") " Nov 01 00:57:26.880782 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880753 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-certs" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.881037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.880992 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-socket" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.881191 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881166 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-certs" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.881417 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881395 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-data" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.881596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881555 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxv6z\" (UniqueName: \"kubernetes.io/projected/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-kube-api-access-fxv6z\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881596 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881585 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881601 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881615 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15803f9b-8f6f-4169-8e00-4b62fcc1e09d-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881647 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881661 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881675 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881689 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j4wb\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-kube-api-access-8j4wb\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881702 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881721 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881735 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5a8676-5f71-4c35-aed8-b373c47c3a9b-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.881902 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.881748 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.882868 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.882391 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-data" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.884564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.883695 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05687ed-54f9-4637-a3d6-451b174895e4-istiod-ca-cert" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:26.884564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.883917 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-credential-socket" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.884564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.884458 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530d5660-94c7-4ae6-a7c4-e825d375d84d-istiod-ca-cert" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:26.884564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.884511 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-socket" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.888803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.888767 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-envoy" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.892773 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.892746 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-credential-socket" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.894890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.894864 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-envoy" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:26.895322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.895015 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a05687ed-54f9-4637-a3d6-451b174895e4-istio-podinfo" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:26.895322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.895051 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-istio-token" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.895322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.895114 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-kube-api-access-fzm6q" pod "a05687ed-54f9-4637-a3d6-451b174895e4" (UID: "a05687ed-54f9-4637-a3d6-451b174895e4"). InnerVolumeSpecName "kube-api-access-fzm6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.895322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.895209 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-podinfo" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:26.895322 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.895284 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-token" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.896538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.896513 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-kube-api-access-2944m" pod "530d5660-94c7-4ae6-a7c4-e825d375d84d" (UID: "530d5660-94c7-4ae6-a7c4-e825d375d84d"). InnerVolumeSpecName "kube-api-access-2944m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:26.984163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984126 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984165 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984182 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984197 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984210 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05687ed-54f9-4637-a3d6-451b174895e4-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984226 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05687ed-54f9-4637-a3d6-451b174895e4-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984240 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2944m\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-kube-api-access-2944m\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984255 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984269 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984282 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzm6q\" (UniqueName: \"kubernetes.io/projected/a05687ed-54f9-4637-a3d6-451b174895e4-kube-api-access-fzm6q\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984295 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530d5660-94c7-4ae6-a7c4-e825d375d84d-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984307 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05687ed-54f9-4637-a3d6-451b174895e4-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984322 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530d5660-94c7-4ae6-a7c4-e825d375d84d-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:26.984408 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:26.984335 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530d5660-94c7-4ae6-a7c4-e825d375d84d-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:27.114429 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.114395 4910 generic.go:356] "Generic (PLEG): container finished" podID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerID="0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05" exitCode=0 Nov 01 00:57:27.114696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.114676 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" event={"ID":"2d5a8676-5f71-4c35-aed8-b373c47c3a9b","Type":"ContainerDied","Data":"0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05"} Nov 01 00:57:27.114803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.114720 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" event={"ID":"2d5a8676-5f71-4c35-aed8-b373c47c3a9b","Type":"ContainerDied","Data":"a276dffc20aa14aa8015234736166a685e50b6d82c14f5280b82d8544042e7ac"} Nov 01 00:57:27.114803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.114741 4910 scope.go:122] "RemoveContainer" containerID="0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05" Nov 01 00:57:27.114910 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.114842 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg" Nov 01 00:57:27.119722 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.119691 4910 generic.go:356] "Generic (PLEG): container finished" podID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerID="676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2" exitCode=0 Nov 01 00:57:27.119875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.119752 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" Nov 01 00:57:27.119875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.119788 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" event={"ID":"15803f9b-8f6f-4169-8e00-4b62fcc1e09d","Type":"ContainerDied","Data":"676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2"} Nov 01 00:57:27.119875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.119847 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz" event={"ID":"15803f9b-8f6f-4169-8e00-4b62fcc1e09d","Type":"ContainerDied","Data":"49569a0d4714d8ac86493fa1d525b0c2b6942d58d5364c9b2eea7c57f735c552"} Nov 01 00:57:27.125449 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.125424 4910 generic.go:356] "Generic (PLEG): container finished" podID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerID="57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400" exitCode=0 Nov 01 00:57:27.125552 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.125474 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" Nov 01 00:57:27.125612 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.125546 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" event={"ID":"530d5660-94c7-4ae6-a7c4-e825d375d84d","Type":"ContainerDied","Data":"57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400"} Nov 01 00:57:27.125612 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.125584 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72" event={"ID":"530d5660-94c7-4ae6-a7c4-e825d375d84d","Type":"ContainerDied","Data":"53b592b8f9b23af444e3ef38d8973b312f68e62b005cd5579577b29871d5666b"} Nov 01 00:57:27.128008 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.127986 4910 generic.go:356] "Generic (PLEG): container finished" podID="a05687ed-54f9-4637-a3d6-451b174895e4" containerID="58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f" exitCode=0 Nov 01 00:57:27.128237 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.128166 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" event={"ID":"a05687ed-54f9-4637-a3d6-451b174895e4","Type":"ContainerDied","Data":"58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f"} Nov 01 00:57:27.129365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.128194 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" Nov 01 00:57:27.139887 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.128396 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26" event={"ID":"a05687ed-54f9-4637-a3d6-451b174895e4","Type":"ContainerDied","Data":"6725b7b28300977ced466887e2f4ed55d84d913f998899a4257f49538ad86d46"} Nov 01 00:57:27.145551 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.145524 4910 scope.go:122] "RemoveContainer" containerID="0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05" Nov 01 00:57:27.145913 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:27.145886 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05\": container with ID starting with 0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05 not found: ID does not exist" containerID="0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05" Nov 01 00:57:27.146075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.146044 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05"} err="failed to get container status \"0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05\": rpc error: code = NotFound desc = could not find container \"0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05\": container with ID starting with 0d011417fc6e1b3d9dd3e286fdc2eceba9644093d0620ae599e0ec0a03c14f05 not found: ID does not exist" Nov 01 00:57:27.146179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.146076 4910 scope.go:122] "RemoveContainer" containerID="676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2" Nov 01 00:57:27.157516 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.157499 4910 scope.go:122] "RemoveContainer" containerID="676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2" Nov 01 00:57:27.157867 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:27.157843 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2\": container with ID starting with 676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2 not found: ID does not exist" containerID="676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2" Nov 01 00:57:27.157978 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.157877 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2"} err="failed to get container status \"676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2\": rpc error: code = NotFound desc = could not find container \"676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2\": container with ID starting with 676a3067cc1e6213125687212a99fcbdafd5603df9b27a88c82c02d762bde8c2 not found: ID does not exist" Nov 01 00:57:27.157978 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.157898 4910 scope.go:122] "RemoveContainer" containerID="57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400" Nov 01 00:57:27.171593 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.171573 4910 scope.go:122] "RemoveContainer" containerID="57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400" Nov 01 00:57:27.171916 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:27.171898 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400\": container with ID starting with 57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400 not found: ID does not exist" containerID="57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400" Nov 01 00:57:27.172000 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.171924 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400"} err="failed to get container status \"57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400\": rpc error: code = NotFound desc = could not find container \"57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400\": container with ID starting with 57c1d7dc3151795806eb24720b7620279b51f2a87b3139e1bfbc10d2bf83e400 not found: ID does not exist" Nov 01 00:57:27.172000 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.171945 4910 scope.go:122] "RemoveContainer" containerID="58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f" Nov 01 00:57:27.182881 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.182864 4910 scope.go:122] "RemoveContainer" containerID="58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f" Nov 01 00:57:27.183140 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:27.183120 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f\": container with ID starting with 58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f not found: ID does not exist" containerID="58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f" Nov 01 00:57:27.183228 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.183145 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f"} err="failed to get container status \"58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f\": rpc error: code = NotFound desc = could not find container \"58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f\": container with ID starting with 58e41d3838fc7deae00c3a0e96c28b517dbe54561c4897b4833a297bb22eca4f not found: ID does not exist" Nov 01 00:57:27.346107 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.346078 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72"] Nov 01 00:57:27.350175 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.350150 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-9rq72"] Nov 01 00:57:27.454420 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.454346 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg"] Nov 01 00:57:27.459712 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.459688 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-hw4wg"] Nov 01 00:57:27.570249 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.570217 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz"] Nov 01 00:57:27.575006 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.574979 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-vj4sz"] Nov 01 00:57:27.686179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.686150 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26"] Nov 01 00:57:27.692752 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:27.692727 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-30fb19-openshift-default-58c485b85-rrz26"] Nov 01 00:57:28.195741 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:28.195705 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" path="/var/lib/kubelet/pods/15803f9b-8f6f-4169-8e00-4b62fcc1e09d/volumes" Nov 01 00:57:28.196463 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:28.196442 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" path="/var/lib/kubelet/pods/2d5a8676-5f71-4c35-aed8-b373c47c3a9b/volumes" Nov 01 00:57:28.197218 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:28.197197 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" path="/var/lib/kubelet/pods/530d5660-94c7-4ae6-a7c4-e825d375d84d/volumes" Nov 01 00:57:28.198269 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:28.198247 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" path="/var/lib/kubelet/pods/a05687ed-54f9-4637-a3d6-451b174895e4/volumes" Nov 01 00:57:31.526705 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.526673 4910 scope.go:122] "RemoveContainer" containerID="8643973d966c3a4e73f604d6e1686d65ea80c001b92f498e085c33b9b14122a6" Nov 01 00:57:31.537777 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.537760 4910 scope.go:122] "RemoveContainer" containerID="0dd5778da99d0faf2d8a5b323896ea328bcabcc26210c139fff92480a56e2a22" Nov 01 00:57:31.547775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.547749 4910 scope.go:122] "RemoveContainer" containerID="51d884f7e85e5af09e9716e18906fa956e416acc659eeec6aea98b002bd3b292" Nov 01 00:57:31.559684 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.559659 4910 scope.go:122] "RemoveContainer" containerID="650f898e085aa868e61ea1e26b1ad1fdd1ffa8290eabc42cf739f1fdb6986926" Nov 01 00:57:31.571668 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.571647 4910 scope.go:122] "RemoveContainer" containerID="8f8aaa591ed7155e9511ec008ce5ae0a760f434b516d8d89c2c9fd55f0307eeb" Nov 01 00:57:31.583688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.583668 4910 scope.go:122] "RemoveContainer" containerID="69cf5614bee4d337b65ddfe9a5969ff5de630c9319cd9417671ea63bb928eef8" Nov 01 00:57:31.593316 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.593298 4910 scope.go:122] "RemoveContainer" containerID="6a85e0ca8f99d8e6adbe2fed5ee0334e87666849beafe5b5a693afc37c682f49" Nov 01 00:57:31.604310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.604286 4910 scope.go:122] "RemoveContainer" containerID="55febcb4e354476b5891fd920f7a3c7804ff96b4f93c2e4ec6a21d99afc0ffd4" Nov 01 00:57:31.614742 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.614704 4910 scope.go:122] "RemoveContainer" containerID="7227751ab23047152b778b53a36f2f0064a51ad6b0f2ec45d4756d6e03d466c1" Nov 01 00:57:31.623837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:31.623817 4910 scope.go:122] "RemoveContainer" containerID="4934af0c6fd5398fd88d9ccf2e7fcd45eedbaea8ad3ad19b9c8c78ca0b44662d" Nov 01 00:57:36.049376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:36.049341 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg"] Nov 01 00:57:36.098903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:36.098862 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92cc638f-jlhjg"] Nov 01 00:57:36.196321 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:36.196285 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5f549b-4bf9-495e-9bcb-3592c0b7be83" path="/var/lib/kubelet/pods/5c5f549b-4bf9-495e-9bcb-3592c0b7be83/volumes" Nov 01 00:57:40.390358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:40.390323 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz"] Nov 01 00:57:40.391169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:40.391139 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" containerID="cri-o://454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185" gracePeriod=30 Nov 01 00:57:43.110047 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:43.110007 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:57:44.315361 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.315283 4910 generic.go:356] "Generic (PLEG): container finished" podID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerID="34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b" exitCode=0 Nov 01 00:57:44.315818 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.315365 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" event={"ID":"c3e9c1df-cc91-4aed-8e1a-eb51042c031d","Type":"ContainerDied","Data":"34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b"} Nov 01 00:57:44.315870 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.315850 4910 scope.go:122] "RemoveContainer" containerID="34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b" Nov 01 00:57:44.317678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.317618 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:44.317764 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.317728 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:44.317803 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.317764 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:57:44.429678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.429645 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:57:44.429903 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:44.429877 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:57:45.325811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.325780 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" event={"ID":"c3e9c1df-cc91-4aed-8e1a-eb51042c031d","Type":"ContainerStarted","Data":"1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec"} Nov 01 00:57:45.849386 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.849350 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:57:45.996701 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.996674 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-credential-socket\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.996892 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.996734 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-podinfo\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.996892 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.996886 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-kube-api-access-9vkfn\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-kube-api-access-9vkfn\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.997015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.996905 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-token\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-token\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.997015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.996950 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-data\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.997015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.996973 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-certs\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.997015 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.996998 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istiod-ca-cert\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.997225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997002 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-credential-socket" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.997225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997061 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-envoy\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.997225 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997097 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-socket\") pod \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\" (UID: \"acfdd122-d2a5-4e1f-8955-a64b56cf435a\") " Nov 01 00:57:45.997434 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997410 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-certs" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.997614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997595 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-socket" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:45.997746 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997726 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.997813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997752 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:45.997874 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997859 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istiod-ca-cert" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:57:45.997968 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:45.997949 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-data" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:46.000599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.000567 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-token" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:46.000748 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.000681 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-envoy" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:57:46.000817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.000746 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-kube-api-access-9vkfn" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "kube-api-access-9vkfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:57:46.000817 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.000774 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-podinfo" pod "acfdd122-d2a5-4e1f-8955-a64b56cf435a" (UID: "acfdd122-d2a5-4e1f-8955-a64b56cf435a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:57:46.098927 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.098897 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:46.098927 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.098926 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vkfn\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-kube-api-access-9vkfn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:46.099132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.098940 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:46.099132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.098948 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:46.099132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.098958 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:46.099132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.098966 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:46.099132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.098976 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acfdd122-d2a5-4e1f-8955-a64b56cf435a-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:57:46.248315 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.248234 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:57:46.248483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.248433 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 00:57:46.248558 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.248486 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 00:57:46.466489 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.466453 4910 generic.go:356] "Generic (PLEG): container finished" podID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerID="454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185" exitCode=0 Nov 01 00:57:46.476647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.468358 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" Nov 01 00:57:46.491696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.491381 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" event={"ID":"acfdd122-d2a5-4e1f-8955-a64b56cf435a","Type":"ContainerDied","Data":"454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185"} Nov 01 00:57:46.491696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.491480 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz" event={"ID":"acfdd122-d2a5-4e1f-8955-a64b56cf435a","Type":"ContainerDied","Data":"6813a404bac7bab6937341535fc9b65a876f8eed0ec0a038a030f9a717a959fb"} Nov 01 00:57:46.491696 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.491514 4910 scope.go:122] "RemoveContainer" containerID="454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185" Nov 01 00:57:46.614727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.614697 4910 scope.go:122] "RemoveContainer" containerID="454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185" Nov 01 00:57:46.616199 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:57:46.616169 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185\": container with ID starting with 454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185 not found: ID does not exist" containerID="454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185" Nov 01 00:57:46.616406 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.616380 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185"} err="failed to get container status \"454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185\": rpc error: code = NotFound desc = could not find container \"454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185\": container with ID starting with 454ddbdbf8155a8564ba2200f9621ad1c6fd46fa3af54f78fd704b90348c4185 not found: ID does not exist" Nov 01 00:57:46.684048 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.684015 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz"] Nov 01 00:57:46.729907 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:46.729877 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-9n2dz"] Nov 01 00:57:47.248636 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:47.248589 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 00:57:47.248811 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:47.248660 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 00:57:48.193760 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:48.193721 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" path="/var/lib/kubelet/pods/acfdd122-d2a5-4e1f-8955-a64b56cf435a/volumes" Nov 01 00:57:48.248324 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:48.248288 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 00:57:48.248495 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:48.248346 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 00:57:49.248196 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:49.248158 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 00:57:49.248707 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:49.248225 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 00:57:50.248886 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:50.248856 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 01 00:57:50.249277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:50.248904 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 01 00:57:51.253273 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:51.253244 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:57:51.253726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:51.253658 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:57:51.254366 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:51.254345 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 00:57:57.698038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:57.698004 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn"] Nov 01 00:57:57.747228 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:57.747195 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a31bcdff-fmbmn"] Nov 01 00:57:58.196788 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:57:58.196754 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f807af07-0df2-4d77-bd2d-8c0b5f10413b" path="/var/lib/kubelet/pods/f807af07-0df2-4d77-bd2d-8c0b5f10413b/volumes" Nov 01 00:58:02.118837 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.118805 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9"] Nov 01 00:58:02.119457 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119444 4910 cpu_manager.go:414] "containerMap: removing container" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" Nov 01 00:58:02.119510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119458 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" Nov 01 00:58:02.119510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119477 4910 cpu_manager.go:414] "containerMap: removing container" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" Nov 01 00:58:02.119510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119482 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" Nov 01 00:58:02.119510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119497 4910 cpu_manager.go:414] "containerMap: removing container" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" Nov 01 00:58:02.119510 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119502 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119515 4910 cpu_manager.go:414] "containerMap: removing container" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119519 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119527 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119531 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119542 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f807af07-0df2-4d77-bd2d-8c0b5f10413b" containerName="revoke-keys" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119546 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f807af07-0df2-4d77-bd2d-8c0b5f10413b" containerName="revoke-keys" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119560 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119565 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119571 4910 cpu_manager.go:414] "containerMap: removing container" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119575 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119691 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05687ed-54f9-4637-a3d6-451b174895e4" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119701 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2972c8-7d7d-4eee-9032-54210747636a" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119712 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="15803f9b-8f6f-4169-8e00-4b62fcc1e09d" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119719 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="530d5660-94c7-4ae6-a7c4-e825d375d84d" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119728 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfdd122-d2a5-4e1f-8955-a64b56cf435a" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119736 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="309de2b1-bd96-45e9-9e5c-5649dede132d" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119742 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5a8676-5f71-4c35-aed8-b373c47c3a9b" containerName="istio-proxy" Nov 01 00:58:02.119739 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.119749 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f807af07-0df2-4d77-bd2d-8c0b5f10413b" containerName="revoke-keys" Nov 01 00:58:02.123796 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.123780 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.177017 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.176972 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 00:58:02.181927 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.181907 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9"] Nov 01 00:58:02.226053 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.226014 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 00:58:02.237968 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.237938 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.238121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.237988 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5q7\" (UniqueName: \"kubernetes.io/projected/f5089845-e580-4c70-8add-bf575c69ac98-kube-api-access-fq5q7\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.238121 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.238080 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.339573 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.339532 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.339979 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.339958 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5q7\" (UniqueName: \"kubernetes.io/projected/f5089845-e580-4c70-8add-bf575c69ac98-kube-api-access-fq5q7\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.340042 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.340031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.340688 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.340664 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.342254 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.342235 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.401200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.401124 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5q7\" (UniqueName: \"kubernetes.io/projected/f5089845-e580-4c70-8add-bf575c69ac98-kube-api-access-fq5q7\") pod \"maas-api-revoke-sub-6cbb1847-77zh9\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.437095 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.437063 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:02.834333 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:58:02.834306 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5089845_e580_4c70_8add_bf575c69ac98.slice/crio-61142eccbdf0043fd61bf55c7912b5a540110fec863fbd19b0e3618252138256 WatchSource:0}: Error finding container 61142eccbdf0043fd61bf55c7912b5a540110fec863fbd19b0e3618252138256: Status 404 returned error can't find the container with id 61142eccbdf0043fd61bf55c7912b5a540110fec863fbd19b0e3618252138256 Nov 01 00:58:02.835529 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:02.835500 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9"] Nov 01 00:58:03.690808 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:03.690769 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" event={"ID":"f5089845-e580-4c70-8add-bf575c69ac98","Type":"ContainerStarted","Data":"b3aaf8d9f6a728ec5eacda825e2d8d2d1c2e466fdc7e89538a1381599519ee20"} Nov 01 00:58:03.691344 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:03.690829 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" event={"ID":"f5089845-e580-4c70-8add-bf575c69ac98","Type":"ContainerStarted","Data":"61142eccbdf0043fd61bf55c7912b5a540110fec863fbd19b0e3618252138256"} Nov 01 00:58:03.808875 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:03.808825 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" podStartSLOduration=1.808810276 podStartE2EDuration="1.808810276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:58:03.807004371 +0000 UTC m=+2073.786586720" watchObservedRunningTime="2026-11-01 00:58:03.808810276 +0000 UTC m=+2073.788392612" Nov 01 00:58:04.702991 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:04.702953 4910 generic.go:356] "Generic (PLEG): container finished" podID="f5089845-e580-4c70-8add-bf575c69ac98" containerID="b3aaf8d9f6a728ec5eacda825e2d8d2d1c2e466fdc7e89538a1381599519ee20" exitCode=0 Nov 01 00:58:04.703390 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:04.703036 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" event={"ID":"f5089845-e580-4c70-8add-bf575c69ac98","Type":"ContainerDied","Data":"b3aaf8d9f6a728ec5eacda825e2d8d2d1c2e466fdc7e89538a1381599519ee20"} Nov 01 00:58:04.908450 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:04.908404 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn"] Nov 01 00:58:04.909161 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:04.909108 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" containerID="cri-o://a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a" gracePeriod=30 Nov 01 00:58:05.896016 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:05.895985 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:06.004840 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.004814 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-xks-ca\") pod \"f5089845-e580-4c70-8add-bf575c69ac98\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " Nov 01 00:58:06.005045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.004944 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5089845-e580-4c70-8add-bf575c69ac98-kube-api-access-fq5q7\" (UniqueName: \"kubernetes.io/projected/f5089845-e580-4c70-8add-bf575c69ac98-kube-api-access-fq5q7\") pod \"f5089845-e580-4c70-8add-bf575c69ac98\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " Nov 01 00:58:06.005045 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.005041 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-openshift-ca\") pod \"f5089845-e580-4c70-8add-bf575c69ac98\" (UID: \"f5089845-e580-4c70-8add-bf575c69ac98\") " Nov 01 00:58:06.005376 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.005353 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-openshift-ca" pod "f5089845-e580-4c70-8add-bf575c69ac98" (UID: "f5089845-e580-4c70-8add-bf575c69ac98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:58:06.005461 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.005424 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:06.007132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.007111 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5089845-e580-4c70-8add-bf575c69ac98-kube-api-access-fq5q7" pod "f5089845-e580-4c70-8add-bf575c69ac98" (UID: "f5089845-e580-4c70-8add-bf575c69ac98"). InnerVolumeSpecName "kube-api-access-fq5q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:58:06.007221 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.007145 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-xks-ca" pod "f5089845-e580-4c70-8add-bf575c69ac98" (UID: "f5089845-e580-4c70-8add-bf575c69ac98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 00:58:06.106661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.106612 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5089845-e580-4c70-8add-bf575c69ac98-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:06.106661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.106662 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq5q7\" (UniqueName: \"kubernetes.io/projected/f5089845-e580-4c70-8add-bf575c69ac98-kube-api-access-fq5q7\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:06.431038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.430948 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l"] Nov 01 00:58:06.431683 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.431664 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f5089845-e580-4c70-8add-bf575c69ac98" containerName="revoke-keys" Nov 01 00:58:06.431807 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.431685 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5089845-e580-4c70-8add-bf575c69ac98" containerName="revoke-keys" Nov 01 00:58:06.431889 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.431875 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5089845-e580-4c70-8add-bf575c69ac98" containerName="revoke-keys" Nov 01 00:58:06.435544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.435511 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.518966 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.518009 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f97e89-openshift-default-dockercfg-75zjq\"" Nov 01 00:58:06.528270 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.524320 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l"] Nov 01 00:58:06.755388 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.755350 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-socket\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.755862 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.755831 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-token\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.756088 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.756061 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8crc\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-kube-api-access-v8crc\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.756222 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.756208 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf8936-fd4f-4915-acf0-c4cdabc12636-istiod-ca-cert\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.756379 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.756366 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-certs\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.756523 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.756511 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-envoy\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.756661 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.756645 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-credential-socket\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.756807 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.756789 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-data\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.757032 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.757003 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-podinfo\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.789214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.788862 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" Nov 01 00:58:06.789214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.788968 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9" event={"ID":"f5089845-e580-4c70-8add-bf575c69ac98","Type":"ContainerDied","Data":"61142eccbdf0043fd61bf55c7912b5a540110fec863fbd19b0e3618252138256"} Nov 01 00:58:06.789214 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.789038 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61142eccbdf0043fd61bf55c7912b5a540110fec863fbd19b0e3618252138256" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.863706 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-envoy\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.863759 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-credential-socket\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.863798 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-data\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.863841 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-podinfo\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.864015 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-socket\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.864083 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-token\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.864122 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8crc\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-kube-api-access-v8crc\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.864155 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf8936-fd4f-4915-acf0-c4cdabc12636-istiod-ca-cert\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.864236 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-certs\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.864749 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-socket\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.864797 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-certs\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.865001 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-data\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.865726 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-credential-socket\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.870649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.867190 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf8936-fd4f-4915-acf0-c4cdabc12636-istiod-ca-cert\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.872961 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.872912 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-podinfo\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.873233 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.873211 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-envoy\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.927649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.927561 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8crc\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-kube-api-access-v8crc\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:06.928131 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:06.928007 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-token\") pod \"e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:07.053992 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.053900 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:07.571158 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:58:07.571126 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cf8936_fd4f_4915_acf0_c4cdabc12636.slice/crio-45fa72ba6a33e18ef18e8f2c178b65dbd6c1a39a4644ec55e8db7a9cd6a21398 WatchSource:0}: Error finding container 45fa72ba6a33e18ef18e8f2c178b65dbd6c1a39a4644ec55e8db7a9cd6a21398: Status 404 returned error can't find the container with id 45fa72ba6a33e18ef18e8f2c178b65dbd6c1a39a4644ec55e8db7a9cd6a21398 Nov 01 00:58:07.571310 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.571156 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l"] Nov 01 00:58:07.574409 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.574369 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:07.574525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.574461 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:07.574525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.574504 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:07.805549 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.805520 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" event={"ID":"51cf8936-fd4f-4915-acf0-c4cdabc12636","Type":"ContainerStarted","Data":"4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018"} Nov 01 00:58:07.805887 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.805804 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" event={"ID":"51cf8936-fd4f-4915-acf0-c4cdabc12636","Type":"ContainerStarted","Data":"45fa72ba6a33e18ef18e8f2c178b65dbd6c1a39a4644ec55e8db7a9cd6a21398"} Nov 01 00:58:07.927883 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:07.927782 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podStartSLOduration=1.9277642849999999 podStartE2EDuration="1.927764285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:58:07.925501569 +0000 UTC m=+2077.905083904" watchObservedRunningTime="2026-11-01 00:58:07.927764285 +0000 UTC m=+2077.907346620" Nov 01 00:58:08.058647 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:08.056229 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:08.062148 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:08.060745 4910 patch_prober.go:29] interesting pod/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 00:58:08.062148 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:08.060802 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 00:58:09.060769 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:09.060738 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:09.822761 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:09.822728 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:09.823848 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:09.823827 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 00:58:10.255532 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.255506 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:58:10.324775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.324745 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-token\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-token\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.324980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.324795 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-socket\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.324980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.324818 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istiod-ca-cert\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.324980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.324885 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-kube-api-access-xfqfn\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-kube-api-access-xfqfn\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.324980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.324937 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-envoy\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.324980 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.324972 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-certs\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.325267 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325006 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-credential-socket\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.325267 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325035 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-data\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.325267 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325065 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-socket" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:58:10.325267 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325136 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-podinfo\") pod \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\" (UID: \"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6\") " Nov 01 00:58:10.325578 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325310 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-certs" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:58:10.325578 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istiod-ca-cert" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:58:10.326001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325889 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-data" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:58:10.326001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325903 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.326001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325926 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.326001 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.325940 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.326277 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.326026 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-credential-socket" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:58:10.327574 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.327547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-envoy" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:58:10.327727 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.327706 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-podinfo" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:58:10.328009 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.327991 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-token" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:58:10.328112 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.328096 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-kube-api-access-xfqfn" pod "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" (UID: "4a43c0dc-f3bc-4b23-a52b-9b548d3767b6"). InnerVolumeSpecName "kube-api-access-xfqfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:58:10.427132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.427030 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.427132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.427071 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfqfn\" (UniqueName: \"kubernetes.io/projected/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-kube-api-access-xfqfn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.427132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.427086 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.427132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.427100 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.427132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.427113 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.427132 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.427127 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:58:10.832035 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.832004 4910 generic.go:356] "Generic (PLEG): container finished" podID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerID="a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a" exitCode=0 Nov 01 00:58:10.832236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.832059 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" Nov 01 00:58:10.832236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.832059 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" event={"ID":"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6","Type":"ContainerDied","Data":"a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a"} Nov 01 00:58:10.832236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.832129 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn" event={"ID":"4a43c0dc-f3bc-4b23-a52b-9b548d3767b6","Type":"ContainerDied","Data":"be7f3ceac11011e96f1b492f9d80e3fe45baacc75634f4318d16c0df4ec84e7f"} Nov 01 00:58:10.832236 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.832168 4910 scope.go:122] "RemoveContainer" containerID="a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a" Nov 01 00:58:10.844899 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.844876 4910 scope.go:122] "RemoveContainer" containerID="08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b" Nov 01 00:58:10.854942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.854925 4910 scope.go:122] "RemoveContainer" containerID="a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a" Nov 01 00:58:10.855255 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:58:10.855227 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a\": container with ID starting with a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a not found: ID does not exist" containerID="a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a" Nov 01 00:58:10.855377 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.855265 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a"} err="failed to get container status \"a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a\": rpc error: code = NotFound desc = could not find container \"a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a\": container with ID starting with a9756da707fbe49ca52b68b827da034efa64346a466bd64f7f1b5f1b9117ad7a not found: ID does not exist" Nov 01 00:58:10.855377 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.855291 4910 scope.go:122] "RemoveContainer" containerID="08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b" Nov 01 00:58:10.855571 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:58:10.855548 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b\": container with ID starting with 08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b not found: ID does not exist" containerID="08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b" Nov 01 00:58:10.855650 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:10.855578 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b"} err="failed to get container status \"08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b\": rpc error: code = NotFound desc = could not find container \"08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b\": container with ID starting with 08d833252b9bdef2e952b5a02f95dc7a95a2a3cc9ec2af9aae6b29214d2aa71b not found: ID does not exist" Nov 01 00:58:11.003942 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:11.003909 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn"] Nov 01 00:58:11.053945 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:11.053908 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a31bcdff-openshift-default-6d68d4bc69-kdrfn"] Nov 01 00:58:12.203693 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:12.200690 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" path="/var/lib/kubelet/pods/4a43c0dc-f3bc-4b23-a52b-9b548d3767b6/volumes" Nov 01 00:58:13.129979 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:58:13.129928 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:58:18.059878 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:18.059840 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc"] Nov 01 00:58:18.109527 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:18.109486 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad05b88d-xvrbc"] Nov 01 00:58:18.196544 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:18.196508 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325a6818-cedf-4573-b3fe-843afefbca0d" path="/var/lib/kubelet/pods/325a6818-cedf-4573-b3fe-843afefbca0d/volumes" Nov 01 00:58:20.851488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.851456 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt"] Nov 01 00:58:20.852111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.852095 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" Nov 01 00:58:20.852151 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.852112 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" Nov 01 00:58:20.852188 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.852159 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" Nov 01 00:58:20.852188 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.852164 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" Nov 01 00:58:20.852270 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.852261 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" Nov 01 00:58:20.852309 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.852275 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a43c0dc-f3bc-4b23-a52b-9b548d3767b6" containerName="istio-proxy" Nov 01 00:58:20.857307 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.857291 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:20.915267 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.915237 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt"] Nov 01 00:58:20.959235 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.959196 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b112c158\"/\"kube-root-ca.crt\"" Nov 01 00:58:20.959473 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.959451 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b112c158\"/\"openshift-service-ca.crt\"" Nov 01 00:58:20.959559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.959477 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b112c158\"/\"default-dockercfg-48k7c\"" Nov 01 00:58:20.960072 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:20.960039 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b112c158\"/\"e2e-lifecycle-model-b112c158-kserve-self-signed-certs\"" Nov 01 00:58:21.025729 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.025687 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-dshm\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.025939 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.025792 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-home\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.025939 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.025851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7253215-068b-424e-be32-4a7654cf124b-tls-certs\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.025939 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.025895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-tmp-dir\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.026102 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.026066 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-model-cache\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.026158 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.026124 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlg6n\" (UniqueName: \"kubernetes.io/projected/d7253215-068b-424e-be32-4a7654cf124b-kube-api-access-dlg6n\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.127965 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.127875 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlg6n\" (UniqueName: \"kubernetes.io/projected/d7253215-068b-424e-be32-4a7654cf124b-kube-api-access-dlg6n\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128012 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-dshm\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128038 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-home\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128065 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7253215-068b-424e-be32-4a7654cf124b-tls-certs\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128155 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128092 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-tmp-dir\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128387 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128205 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-model-cache\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128513 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128489 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-home\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128563 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-tmp-dir\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.128690 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.128595 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-model-cache\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.130597 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.130575 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-dshm\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.130820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.130801 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7253215-068b-424e-be32-4a7654cf124b-tls-certs\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.189890 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.189863 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlg6n\" (UniqueName: \"kubernetes.io/projected/d7253215-068b-424e-be32-4a7654cf124b-kube-api-access-dlg6n\") pod \"e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.470210 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.470121 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:21.861266 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:58:21.861233 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7253215_068b_424e_be32_4a7654cf124b.slice/crio-a4df6a9de53579a75503256fcba7bfab384009a9b7495f87f7b9736f7c8cd6ae WatchSource:0}: Error finding container a4df6a9de53579a75503256fcba7bfab384009a9b7495f87f7b9736f7c8cd6ae: Status 404 returned error can't find the container with id a4df6a9de53579a75503256fcba7bfab384009a9b7495f87f7b9736f7c8cd6ae Nov 01 00:58:21.861749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.861727 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt"] Nov 01 00:58:21.933039 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:21.933008 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" event={"ID":"d7253215-068b-424e-be32-4a7654cf124b","Type":"ContainerStarted","Data":"a4df6a9de53579a75503256fcba7bfab384009a9b7495f87f7b9736f7c8cd6ae"} Nov 01 00:58:22.052215 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:22.052153 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" podStartSLOduration=2.052137039 podStartE2EDuration="2.052137039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:58:22.050761123 +0000 UTC m=+2092.030343460" watchObservedRunningTime="2026-11-01 00:58:22.052137039 +0000 UTC m=+2092.031719374" Nov 01 00:58:22.470481 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:22.470452 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:22.942559 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:22.942519 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" event={"ID":"d7253215-068b-424e-be32-4a7654cf124b","Type":"ContainerStarted","Data":"8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1"} Nov 01 00:58:30.663775 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:30.663743 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:58:30.669044 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:30.669020 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 00:58:30.675033 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:30.675009 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:58:30.680024 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:30.680004 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 00:58:31.470348 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:31.470300 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:31.483800 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:31.483775 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:31.740110 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:31.740039 4910 scope.go:122] "RemoveContainer" containerID="53a6902290c965978be00f598ff00feae78c18d8541cdb636637ef8a1511fdc9" Nov 01 00:58:31.750564 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:31.750546 4910 scope.go:122] "RemoveContainer" containerID="29abb3aa0e916e54751725a73f5bc57b7f9d5af6c0575ae5a1dcaa34ccdccc96" Nov 01 00:58:31.766038 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:31.766006 4910 scope.go:122] "RemoveContainer" containerID="405f2d6115f25571c3b38d1e3cfac89e906de2fa2956b0018a32d603126f2d89" Nov 01 00:58:32.033433 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:32.033408 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 00:58:38.022649 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:38.016743 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 00:58:43.111807 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:58:43.111729 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:58:58.542606 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.542571 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4"] Nov 01 00:58:58.547171 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.547152 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.633588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.633559 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e296f9-openshift-default-dockercfg-52z6k\"" Nov 01 00:58:58.670358 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.669598 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4"] Nov 01 00:58:58.680185 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.680148 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4nnz\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-kube-api-access-g4nnz\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.680432 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.680411 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-data\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.680562 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.680546 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266e279e-d94f-481c-af70-1c0eb92163d4-istio-podinfo\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.680783 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.680763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-envoy\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.681086 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.680909 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266e279e-d94f-481c-af70-1c0eb92163d4-istiod-ca-cert\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.681318 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.681299 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-socket\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.681465 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.681450 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-credential-socket\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.681580 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.681566 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-istio-token\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.681824 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.681806 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-certs\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.835309 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.835211 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-data\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.846920 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.841117 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-data\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.856268 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.847185 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266e279e-d94f-481c-af70-1c0eb92163d4-istio-podinfo\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.856525 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.856500 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-envoy\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.878609 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.878532 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266e279e-d94f-481c-af70-1c0eb92163d4-istiod-ca-cert\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.878969 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.878950 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266e279e-d94f-481c-af70-1c0eb92163d4-istiod-ca-cert\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.897229 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.897183 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-socket\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.898017 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.897995 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-credential-socket\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.898200 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.898183 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-istio-token\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.924404 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.898974 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-credential-socket\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.924894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.924870 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-certs\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.925296 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.925276 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-socket\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.925813 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.925782 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4nnz\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-kube-api-access-g4nnz\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.926849 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.926300 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-certs\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.930613 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.930580 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-envoy\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.962795 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.962767 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-istio-token\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.983047 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.982954 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266e279e-d94f-481c-af70-1c0eb92163d4-istio-podinfo\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:58.999217 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:58.999152 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4nnz\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-kube-api-access-g4nnz\") pod \"e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:59.164304 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:59.164217 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:58:59.664031 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:59.664000 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4"] Nov 01 00:58:59.664475 ip-10-0-137-150 kubenswrapper[4910]: W1101 00:58:59.664432 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266e279e_d94f_481c_af70_1c0eb92163d4.slice/crio-1e6c026213634dee9bdda9d929ab28a5cf9de319071b847ea292aa72dd50118e WatchSource:0}: Error finding container 1e6c026213634dee9bdda9d929ab28a5cf9de319071b847ea292aa72dd50118e: Status 404 returned error can't find the container with id 1e6c026213634dee9bdda9d929ab28a5cf9de319071b847ea292aa72dd50118e Nov 01 00:58:59.667349 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:59.667316 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:59.667423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:59.667380 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:58:59.667423 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:58:59.667412 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:00.555891 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:00.555836 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" event={"ID":"266e279e-d94f-481c-af70-1c0eb92163d4","Type":"ContainerStarted","Data":"1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669"} Nov 01 00:59:00.556080 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:00.555927 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" event={"ID":"266e279e-d94f-481c-af70-1c0eb92163d4","Type":"ContainerStarted","Data":"1e6c026213634dee9bdda9d929ab28a5cf9de319071b847ea292aa72dd50118e"} Nov 01 00:59:00.729730 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:00.729664 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podStartSLOduration=2.72961614 podStartE2EDuration="2.72961614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:00.726187171 +0000 UTC m=+2130.705769508" watchObservedRunningTime="2026-11-01 00:59:00.72961614 +0000 UTC m=+2130.709198474" Nov 01 00:59:01.164818 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:01.164783 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:59:01.171487 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:01.171462 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:59:01.564111 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:01.564079 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:59:01.565186 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:01.565169 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 00:59:08.474518 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.474482 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9"] Nov 01 00:59:08.479069 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.479051 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.585656 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.584696 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-bc241b-openshift-default-dockercfg-ch9vv\"" Nov 01 00:59:08.755749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.739276 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9"] Nov 01 00:59:08.911365 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.911326 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-data\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.911749 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.911725 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-certs\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.911927 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.911912 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-socket\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.912040 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.912028 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-envoy\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.912133 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.912121 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-token\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.912329 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.912314 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdx6\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-kube-api-access-6xdx6\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.912496 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.912483 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-credential-socket\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.912678 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.912662 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-podinfo\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:08.912820 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:08.912804 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44489bec-b295-4631-ae02-50ed5f97a2cd-istiod-ca-cert\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.013894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.013808 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdx6\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-kube-api-access-6xdx6\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.013904 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-credential-socket\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.013973 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-podinfo\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.014008 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44489bec-b295-4631-ae02-50ed5f97a2cd-istiod-ca-cert\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.014063 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-data\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014169 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.014160 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-certs\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.014215 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-socket\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.014252 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-envoy\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.014440 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.014278 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-token\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.015158 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.015119 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-credential-socket\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.015342 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.015314 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-data\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.015431 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.015371 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-certs\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.015613 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.015596 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-socket\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.016908 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.016883 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44489bec-b295-4631-ae02-50ed5f97a2cd-istiod-ca-cert\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.018260 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.018238 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-envoy\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.018726 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.018702 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-podinfo\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.076017 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.075980 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-token\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.076179 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.076090 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdx6\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-kube-api-access-6xdx6\") pod \"e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.103821 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.103795 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:09.813163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.813058 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:09.813768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.813172 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:09.813768 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.813220 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 00:59:09.826676 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.825681 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9"] Nov 01 00:59:09.861394 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:09.861359 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" event={"ID":"44489bec-b295-4631-ae02-50ed5f97a2cd","Type":"ContainerStarted","Data":"d945642915e416beca0f13aa86c781616e5fcc73610691fd10e7c2dc64d2179b"} Nov 01 00:59:10.874003 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:10.873954 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" event={"ID":"44489bec-b295-4631-ae02-50ed5f97a2cd","Type":"ContainerStarted","Data":"4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a"} Nov 01 00:59:11.046828 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:11.046754 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podStartSLOduration=3.046734776 podStartE2EDuration="3.046734776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 00:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 00:59:11.045519501 +0000 UTC m=+2141.025101860" watchObservedRunningTime="2026-11-01 00:59:11.046734776 +0000 UTC m=+2141.026317113" Nov 01 00:59:11.104190 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:11.104158 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:11.110075 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:11.110052 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:11.840599 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:11.840566 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg"] Nov 01 00:59:11.840806 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:11.840774 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" containerID="cri-o://6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b" gracePeriod=30 Nov 01 00:59:11.893453 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:11.893423 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:11.895951 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:11.895894 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 00:59:13.104539 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:59:13.104506 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 00:59:15.958842 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:15.958806 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 00:59:15.959488 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:15.959443 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 00:59:17.258685 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.258658 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387172 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-kube-api-access-64q7g\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-kube-api-access-64q7g\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387237 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-certs\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387285 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istiod-ca-cert\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387356 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-token\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-token\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387448 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-data\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387510 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-socket\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387660 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-podinfo\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387770 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-credential-socket\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.388686 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.387865 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-envoy\") pod \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\" (UID: \"b5abcdca-f9e8-4f6a-b644-34df1bfe236b\") " Nov 01 00:59:17.390588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.389743 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-socket" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:59:17.390588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.389891 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-data" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:59:17.390588 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.390542 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-credential-socket" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:59:17.391037 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.391005 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istiod-ca-cert" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 00:59:17.391264 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.391235 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-certs" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:59:17.432112 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.432061 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-podinfo" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 00:59:17.467483 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.467424 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-kube-api-access-64q7g" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "kube-api-access-64q7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:59:17.468163 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.468121 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-envoy" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 00:59:17.468468 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.468417 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-token" pod "b5abcdca-f9e8-4f6a-b644-34df1bfe236b" (UID: "b5abcdca-f9e8-4f6a-b644-34df1bfe236b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512003 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64q7g\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-kube-api-access-64q7g\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512053 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512072 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512087 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512101 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512131 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512146 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512160 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:17.512238 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:17.512175 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5abcdca-f9e8-4f6a-b644-34df1bfe236b-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 00:59:18.014822 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.014784 4910 generic.go:356] "Generic (PLEG): container finished" podID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerID="6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b" exitCode=0 Nov 01 00:59:18.015314 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.015267 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" event={"ID":"b5abcdca-f9e8-4f6a-b644-34df1bfe236b","Type":"ContainerDied","Data":"6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b"} Nov 01 00:59:18.015538 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.015513 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" event={"ID":"b5abcdca-f9e8-4f6a-b644-34df1bfe236b","Type":"ContainerDied","Data":"5425e10ce07e606da95bcf9b469dac3535f8e23b5b86afe88d3c19e988c1ede8"} Nov 01 00:59:18.015703 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.015689 4910 scope.go:122] "RemoveContainer" containerID="6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b" Nov 01 00:59:18.015986 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.015971 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg" Nov 01 00:59:18.039244 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.039216 4910 scope.go:122] "RemoveContainer" containerID="6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b" Nov 01 00:59:18.040093 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:59:18.040068 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b\": container with ID starting with 6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b not found: ID does not exist" containerID="6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b" Nov 01 00:59:18.040199 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.040108 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b"} err="failed to get container status \"6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b\": rpc error: code = NotFound desc = could not find container \"6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b\": container with ID starting with 6c9d21de3bead8241fe89c07fd620dd2a72da858fbe976f70bd76309c593f88b not found: ID does not exist" Nov 01 00:59:18.202840 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.202806 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg"] Nov 01 00:59:18.252414 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:18.252378 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-scdrg"] Nov 01 00:59:20.194614 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:20.194582 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" path="/var/lib/kubelet/pods/b5abcdca-f9e8-4f6a-b644-34df1bfe236b/volumes" Nov 01 00:59:31.827265 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:31.827235 4910 scope.go:122] "RemoveContainer" containerID="08aa0a83e8074c773a6ab59b9427dd5972eea06d86f960a5f502eade3eed7240" Nov 01 00:59:31.838341 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:31.838319 4910 scope.go:122] "RemoveContainer" containerID="1f57f592aaba80dc0387d07bae6df74f15b046d04e46008ace1b067e6d3e2fc9" Nov 01 00:59:31.848894 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:31.848796 4910 scope.go:122] "RemoveContainer" containerID="fa246e0929a725bdc003b0ac25198833202dc47b4a46897f378210fc050fd951" Nov 01 00:59:31.862921 ip-10-0-137-150 kubenswrapper[4910]: I1101 00:59:31.862900 4910 scope.go:122] "RemoveContainer" containerID="5f241662c54750c2fa3609049f6cec54e0ff4c867138a6267028bc8e333c31e6" Nov 01 00:59:43.103912 ip-10-0-137-150 kubenswrapper[4910]: E1101 00:59:43.103874 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:00:00.202656 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.202608 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp"] Nov 01 01:00:00.203210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.203193 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" Nov 01 01:00:00.203257 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.203211 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" Nov 01 01:00:00.203320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.203311 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5abcdca-f9e8-4f6a-b644-34df1bfe236b" containerName="istio-proxy" Nov 01 01:00:00.206340 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.206319 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8"] Nov 01 01:00:00.206453 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.206435 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.211046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.211020 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp"] Nov 01 01:00:00.211154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.211108 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.215319 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.215297 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2"] Nov 01 01:00:00.216280 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.216259 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.219125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.219105 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.224385 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.224368 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr"] Nov 01 01:00:00.230415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.229138 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx"] Nov 01 01:00:00.230415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.229482 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.233825 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.233804 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn"] Nov 01 01:00:00.233963 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.233948 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.238423 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.238405 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w"] Nov 01 01:00:00.238600 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.238462 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.243519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.243504 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25"] Nov 01 01:00:00.243767 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.243569 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.249437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.249417 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr"] Nov 01 01:00:00.249570 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.249554 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:00.288245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.288220 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.289358 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.289337 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp"] Nov 01 01:00:00.301064 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.301041 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8"] Nov 01 01:00:00.312609 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.312582 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2"] Nov 01 01:00:00.320012 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.319898 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqll2\" (UniqueName: \"kubernetes.io/projected/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-kube-api-access-sqll2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.321937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.321906 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe5eb59-798f-4b82-b8fc-459178830ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.322267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.322245 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d97be-9f94-4289-aafe-32330f14ceb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.322567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.322477 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8617a04-4c0e-4ea6-ae6c-75925a6af130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.322789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.322769 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thck\" (UniqueName: \"kubernetes.io/projected/971d1be6-8abd-4920-a123-724c9d972483-kube-api-access-7thck\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.322928 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.322910 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971d1be6-8abd-4920-a123-724c9d972483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.323093 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.323075 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.323328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.323311 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.324808 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.324786 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49sdj\" (UniqueName: \"kubernetes.io/projected/231d97be-9f94-4289-aafe-32330f14ceb2-kube-api-access-49sdj\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.325050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.325030 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7s4\" (UniqueName: \"kubernetes.io/projected/d8617a04-4c0e-4ea6-ae6c-75925a6af130-kube-api-access-4q7s4\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.325269 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.325250 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmndb\" (UniqueName: \"kubernetes.io/projected/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-kube-api-access-mmndb\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.325429 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.325413 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgk9f\" (UniqueName: \"kubernetes.io/projected/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-kube-api-access-wgk9f\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.328377 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.328349 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsnw\" (UniqueName: \"kubernetes.io/projected/1e090764-767d-4f29-bd16-a94cee6b5d82-kube-api-access-wcsnw\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.328549 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.328526 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdzf\" (UniqueName: \"kubernetes.io/projected/ebe5eb59-798f-4b82-b8fc-459178830ba8-kube-api-access-2qdzf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.328963 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.328938 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e090764-767d-4f29-bd16-a94cee6b5d82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.329068 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.329000 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.394822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.367694 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:00:00.411783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.410618 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr"] Nov 01 01:00:00.421290 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.421264 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx"] Nov 01 01:00:00.435183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.435149 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w"] Nov 01 01:00:00.437398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437368 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmndb\" (UniqueName: \"kubernetes.io/projected/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-kube-api-access-mmndb\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.437540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437406 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgk9f\" (UniqueName: \"kubernetes.io/projected/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-kube-api-access-wgk9f\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.437540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437434 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsnw\" (UniqueName: \"kubernetes.io/projected/1e090764-767d-4f29-bd16-a94cee6b5d82-kube-api-access-wcsnw\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.437540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437467 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdzf\" (UniqueName: \"kubernetes.io/projected/ebe5eb59-798f-4b82-b8fc-459178830ba8-kube-api-access-2qdzf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.437540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437496 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e090764-767d-4f29-bd16-a94cee6b5d82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.437540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437517 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437567 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqll2\" (UniqueName: \"kubernetes.io/projected/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-kube-api-access-sqll2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437596 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe5eb59-798f-4b82-b8fc-459178830ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437644 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d97be-9f94-4289-aafe-32330f14ceb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437724 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25x8d\" (UniqueName: \"kubernetes.io/projected/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-kube-api-access-25x8d\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437770 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8617a04-4c0e-4ea6-ae6c-75925a6af130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437828 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thck\" (UniqueName: \"kubernetes.io/projected/971d1be6-8abd-4920-a123-724c9d972483-kube-api-access-7thck\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437861 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971d1be6-8abd-4920-a123-724c9d972483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.437934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437890 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:00.438409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.437952 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.438409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.438004 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.438409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.438067 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49sdj\" (UniqueName: \"kubernetes.io/projected/231d97be-9f94-4289-aafe-32330f14ceb2-kube-api-access-49sdj\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.438409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.438108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw2dz\" (UniqueName: \"kubernetes.io/projected/8a75cd45-43a4-439c-aac7-638366038761-kube-api-access-mw2dz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.438409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.438169 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7s4\" (UniqueName: \"kubernetes.io/projected/d8617a04-4c0e-4ea6-ae6c-75925a6af130-kube-api-access-4q7s4\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.438409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.438235 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a75cd45-43a4-439c-aac7-638366038761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.440222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.440192 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e090764-767d-4f29-bd16-a94cee6b5d82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.441124 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.441037 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.443654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.442215 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe5eb59-798f-4b82-b8fc-459178830ba8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.443654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.442724 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d97be-9f94-4289-aafe-32330f14ceb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.443654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.443481 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8617a04-4c0e-4ea6-ae6c-75925a6af130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.444246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.444159 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.445669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.444437 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.445669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.445101 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971d1be6-8abd-4920-a123-724c9d972483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.446758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.446688 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr"] Nov 01 01:00:00.490797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.481694 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp"] Nov 01 01:00:00.506144 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.506112 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn"] Nov 01 01:00:00.509222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.509190 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqll2\" (UniqueName: \"kubernetes.io/projected/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-kube-api-access-sqll2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.509432 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.509260 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thck\" (UniqueName: \"kubernetes.io/projected/971d1be6-8abd-4920-a123-724c9d972483-kube-api-access-7thck\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.513516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.513189 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdzf\" (UniqueName: \"kubernetes.io/projected/ebe5eb59-798f-4b82-b8fc-459178830ba8-kube-api-access-2qdzf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.514097 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.514076 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgk9f\" (UniqueName: \"kubernetes.io/projected/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-kube-api-access-wgk9f\") pod \"maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.522661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.519449 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25"] Nov 01 01:00:00.524289 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.524244 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsnw\" (UniqueName: \"kubernetes.io/projected/1e090764-767d-4f29-bd16-a94cee6b5d82-kube-api-access-wcsnw\") pod \"maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.532321 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.532301 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmndb\" (UniqueName: \"kubernetes.io/projected/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-kube-api-access-mmndb\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.535449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.535423 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7s4\" (UniqueName: \"kubernetes.io/projected/d8617a04-4c0e-4ea6-ae6c-75925a6af130-kube-api-access-4q7s4\") pod \"maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.537408 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.537360 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49sdj\" (UniqueName: \"kubernetes.io/projected/231d97be-9f94-4289-aafe-32330f14ceb2-kube-api-access-49sdj\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.540449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.539464 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw2dz\" (UniqueName: \"kubernetes.io/projected/8a75cd45-43a4-439c-aac7-638366038761-kube-api-access-mw2dz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.540449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.539564 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a75cd45-43a4-439c-aac7-638366038761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.540449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.539905 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25x8d\" (UniqueName: \"kubernetes.io/projected/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-kube-api-access-25x8d\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:00.540449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.539978 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:00.540449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.540452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a75cd45-43a4-439c-aac7-638366038761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.541095 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.541070 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:00.549286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.549270 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:00.588469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.588315 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:00.611936 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.607269 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:00.619958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.616405 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25x8d\" (UniqueName: \"kubernetes.io/projected/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-kube-api-access-25x8d\") pod \"maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:00.619958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.619016 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw2dz\" (UniqueName: \"kubernetes.io/projected/8a75cd45-43a4-439c-aac7-638366038761-kube-api-access-mw2dz\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.645286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.645262 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:00.677660 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.668703 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:00.689868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.689823 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:00.694725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.694651 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:00.739963 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.739904 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:00.776139 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.775921 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:00.807954 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:00.806979 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:01.284176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.284133 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp"] Nov 01 01:00:01.443275 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.443104 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" event={"ID":"ebe5eb59-798f-4b82-b8fc-459178830ba8","Type":"ContainerStarted","Data":"afa7b4b790f2623a5749e36418b172d3733fa272cb85e36a8e22694c1fec4306"} Nov 01 01:00:01.516613 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.516579 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp"] Nov 01 01:00:01.525314 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.523380 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8"] Nov 01 01:00:01.589089 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:00:01.588915 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4bf6f4_4336_4fa5_a8ed_adae92b6a626.slice/crio-e61f834f3edd7e4bde8d875e0ea396077be1dcbd55350205291ec9aa1c1f8a2c WatchSource:0}: Error finding container e61f834f3edd7e4bde8d875e0ea396077be1dcbd55350205291ec9aa1c1f8a2c: Status 404 returned error can't find the container with id e61f834f3edd7e4bde8d875e0ea396077be1dcbd55350205291ec9aa1c1f8a2c Nov 01 01:00:01.611970 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.611936 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w"] Nov 01 01:00:01.616564 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:00:01.616526 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55cc5a9_f6b9_4055_af1d_ae6eca77ade6.slice/crio-87a216f160fc5c60156ef4d4d2c5584dee501d473b835b498ef8a15079f29bbb WatchSource:0}: Error finding container 87a216f160fc5c60156ef4d4d2c5584dee501d473b835b498ef8a15079f29bbb: Status 404 returned error can't find the container with id 87a216f160fc5c60156ef4d4d2c5584dee501d473b835b498ef8a15079f29bbb Nov 01 01:00:01.625830 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:00:01.625792 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b0d6a5_dbbf_4e24_a185_c24bb3a728b4.slice/crio-12d2450d2b63b189d1217f618d256bf3709dd46f2b01287c203da9444e84587b WatchSource:0}: Error finding container 12d2450d2b63b189d1217f618d256bf3709dd46f2b01287c203da9444e84587b: Status 404 returned error can't find the container with id 12d2450d2b63b189d1217f618d256bf3709dd46f2b01287c203da9444e84587b Nov 01 01:00:01.629137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.629111 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25"] Nov 01 01:00:01.644660 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:00:01.642060 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a75cd45_43a4_439c_aac7_638366038761.slice/crio-59b24f0fc19e9ee0ab0fcb266afc7764fe09724097ec557c26c00fa9df8684f8 WatchSource:0}: Error finding container 59b24f0fc19e9ee0ab0fcb266afc7764fe09724097ec557c26c00fa9df8684f8: Status 404 returned error can't find the container with id 59b24f0fc19e9ee0ab0fcb266afc7764fe09724097ec557c26c00fa9df8684f8 Nov 01 01:00:01.644660 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.644402 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr"] Nov 01 01:00:01.681695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.681667 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2"] Nov 01 01:00:01.695596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.695573 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn"] Nov 01 01:00:01.702885 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.702860 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr"] Nov 01 01:00:01.704871 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:00:01.704840 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971d1be6_8abd_4920_a123_724c9d972483.slice/crio-e323ca72ce44ea013b2c548bf21c00cda85f2b43c84558844ab6b540edad8e60 WatchSource:0}: Error finding container e323ca72ce44ea013b2c548bf21c00cda85f2b43c84558844ab6b540edad8e60: Status 404 returned error can't find the container with id e323ca72ce44ea013b2c548bf21c00cda85f2b43c84558844ab6b540edad8e60 Nov 01 01:00:01.724741 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:01.724401 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx"] Nov 01 01:00:02.491215 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.491187 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerStarted","Data":"0810149aea1e16aa1b66b77a26571bb4b3dfb657839d3b9ad40b8c64d923778f"} Nov 01 01:00:02.491579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.491239 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerStarted","Data":"e61f834f3edd7e4bde8d875e0ea396077be1dcbd55350205291ec9aa1c1f8a2c"} Nov 01 01:00:02.493641 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.493597 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" event={"ID":"8a75cd45-43a4-439c-aac7-638366038761","Type":"ContainerStarted","Data":"898e15516d75948da7f46310b61d7bf107d65adc58f5f69949abbac491fff26c"} Nov 01 01:00:02.493752 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.493660 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" event={"ID":"8a75cd45-43a4-439c-aac7-638366038761","Type":"ContainerStarted","Data":"59b24f0fc19e9ee0ab0fcb266afc7764fe09724097ec557c26c00fa9df8684f8"} Nov 01 01:00:02.495917 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.495849 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" event={"ID":"971d1be6-8abd-4920-a123-724c9d972483","Type":"ContainerStarted","Data":"9db11c82745dd8c50288a838bec32d4bfe15c088bd3d8731548382d96b9d89a3"} Nov 01 01:00:02.495917 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.495889 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" event={"ID":"971d1be6-8abd-4920-a123-724c9d972483","Type":"ContainerStarted","Data":"e323ca72ce44ea013b2c548bf21c00cda85f2b43c84558844ab6b540edad8e60"} Nov 01 01:00:02.497549 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.497528 4910 generic.go:356] "Generic (PLEG): container finished" podID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerID="c07af8633d6ae5b5499f940baf51466ed0fe1d90a5015c3cd4d46b531e1e2579" exitCode=60 Nov 01 01:00:02.497684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.497603 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" event={"ID":"ebe5eb59-798f-4b82-b8fc-459178830ba8","Type":"ContainerDied","Data":"c07af8633d6ae5b5499f940baf51466ed0fe1d90a5015c3cd4d46b531e1e2579"} Nov 01 01:00:02.498054 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.497808 4910 scope.go:122] "RemoveContainer" containerID="c07af8633d6ae5b5499f940baf51466ed0fe1d90a5015c3cd4d46b531e1e2579" Nov 01 01:00:02.500905 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.500880 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" event={"ID":"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6","Type":"ContainerStarted","Data":"81c924074fde81f98700fb9d8b33693466cf0479798f930bf1f86aaad5706a11"} Nov 01 01:00:02.500988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.500924 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" event={"ID":"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6","Type":"ContainerStarted","Data":"87a216f160fc5c60156ef4d4d2c5584dee501d473b835b498ef8a15079f29bbb"} Nov 01 01:00:02.502753 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.502734 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" event={"ID":"1e090764-767d-4f29-bd16-a94cee6b5d82","Type":"ContainerStarted","Data":"4409e5084e6ad00d195cb329fb8fd9399089ddabaa0dbc607f9afa8368f865eb"} Nov 01 01:00:02.504310 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.504292 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" event={"ID":"d8617a04-4c0e-4ea6-ae6c-75925a6af130","Type":"ContainerStarted","Data":"fee4f18fbee778e7c6f2e5aec0dd07b867dd7d02f67e0a6f05ed849964e04cad"} Nov 01 01:00:02.504359 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.504326 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" event={"ID":"d8617a04-4c0e-4ea6-ae6c-75925a6af130","Type":"ContainerStarted","Data":"23452b198f68781f95e5a711d2449756654a194f19d9d5ebb055ce1cc161ae1f"} Nov 01 01:00:02.505959 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.505936 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerStarted","Data":"23513d68362dc68b988516497199c4955afe65030c9cfb97233d6565408d7e2c"} Nov 01 01:00:02.506042 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.505979 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerStarted","Data":"4220d861edc3b31a6af1e3147bb28076af22b799e32b04f4d6d80ef6917027f2"} Nov 01 01:00:02.508010 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.507989 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" event={"ID":"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a","Type":"ContainerStarted","Data":"7e24a4d2024ca638f21efd4dcee255ef038bba453c92bf0affcb2f29fd0dc5c8"} Nov 01 01:00:02.508088 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.508034 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" event={"ID":"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a","Type":"ContainerStarted","Data":"d3073c54f871bdb851a304ba2d6a12b2883bafcc0a0da2f57d919a89c60aa041"} Nov 01 01:00:02.514812 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.514785 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerStarted","Data":"d730dcaa3292ec05d1d9abd6bd5f997a55ed11ecf072fe2b237aaa33c8708371"} Nov 01 01:00:02.514922 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.514832 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerStarted","Data":"12d2450d2b63b189d1217f618d256bf3709dd46f2b01287c203da9444e84587b"} Nov 01 01:00:02.609578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.609521 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" podStartSLOduration=2.6095044549999997 podStartE2EDuration="2.609504455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:02.606835742 +0000 UTC m=+2192.586418079" watchObservedRunningTime="2026-11-01 01:00:02.609504455 +0000 UTC m=+2192.589086791" Nov 01 01:00:02.723210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.723157 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" podStartSLOduration=2.7231437979999997 podStartE2EDuration="2.723143798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:02.722007469 +0000 UTC m=+2192.701589821" watchObservedRunningTime="2026-11-01 01:00:02.723143798 +0000 UTC m=+2192.702726360" Nov 01 01:00:02.840820 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:02.840772 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" podStartSLOduration=2.8407597989999998 podStartE2EDuration="2.840759799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:02.837665202 +0000 UTC m=+2192.817247538" watchObservedRunningTime="2026-11-01 01:00:02.840759799 +0000 UTC m=+2192.820342186" Nov 01 01:00:03.075412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.075356 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" podStartSLOduration=3.075337154 podStartE2EDuration="3.075337154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:02.955884515 +0000 UTC m=+2192.935466852" watchObservedRunningTime="2026-11-01 01:00:03.075337154 +0000 UTC m=+2193.054919489" Nov 01 01:00:03.078745 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.078699 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" podStartSLOduration=3.078682891 podStartE2EDuration="3.078682891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:03.073614587 +0000 UTC m=+2193.053196923" watchObservedRunningTime="2026-11-01 01:00:03.078682891 +0000 UTC m=+2193.058265228" Nov 01 01:00:03.208302 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.208257 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" podStartSLOduration=3.208242757 podStartE2EDuration="3.208242757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:03.208036245 +0000 UTC m=+2193.187618772" watchObservedRunningTime="2026-11-01 01:00:03.208242757 +0000 UTC m=+2193.187825091" Nov 01 01:00:03.442506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.442414 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" podStartSLOduration=3.442397556 podStartE2EDuration="3.442397556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:03.438459093 +0000 UTC m=+2193.418041430" watchObservedRunningTime="2026-11-01 01:00:03.442397556 +0000 UTC m=+2193.421979895" Nov 01 01:00:03.528098 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.528063 4910 generic.go:356] "Generic (PLEG): container finished" podID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerID="d730dcaa3292ec05d1d9abd6bd5f997a55ed11ecf072fe2b237aaa33c8708371" exitCode=60 Nov 01 01:00:03.528539 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.528144 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerDied","Data":"d730dcaa3292ec05d1d9abd6bd5f997a55ed11ecf072fe2b237aaa33c8708371"} Nov 01 01:00:03.528539 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.528462 4910 scope.go:122] "RemoveContainer" containerID="d730dcaa3292ec05d1d9abd6bd5f997a55ed11ecf072fe2b237aaa33c8708371" Nov 01 01:00:03.530181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.530155 4910 generic.go:356] "Generic (PLEG): container finished" podID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerID="0810149aea1e16aa1b66b77a26571bb4b3dfb657839d3b9ad40b8c64d923778f" exitCode=60 Nov 01 01:00:03.530308 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.530198 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerDied","Data":"0810149aea1e16aa1b66b77a26571bb4b3dfb657839d3b9ad40b8c64d923778f"} Nov 01 01:00:03.530503 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.530480 4910 scope.go:122] "RemoveContainer" containerID="0810149aea1e16aa1b66b77a26571bb4b3dfb657839d3b9ad40b8c64d923778f" Nov 01 01:00:03.532167 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.532150 4910 generic.go:356] "Generic (PLEG): container finished" podID="8a75cd45-43a4-439c-aac7-638366038761" containerID="898e15516d75948da7f46310b61d7bf107d65adc58f5f69949abbac491fff26c" exitCode=60 Nov 01 01:00:03.532267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.532245 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" event={"ID":"8a75cd45-43a4-439c-aac7-638366038761","Type":"ContainerDied","Data":"898e15516d75948da7f46310b61d7bf107d65adc58f5f69949abbac491fff26c"} Nov 01 01:00:03.532556 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.532525 4910 scope.go:122] "RemoveContainer" containerID="898e15516d75948da7f46310b61d7bf107d65adc58f5f69949abbac491fff26c" Nov 01 01:00:03.534545 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.534524 4910 generic.go:356] "Generic (PLEG): container finished" podID="971d1be6-8abd-4920-a123-724c9d972483" containerID="9db11c82745dd8c50288a838bec32d4bfe15c088bd3d8731548382d96b9d89a3" exitCode=60 Nov 01 01:00:03.534676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.534644 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" event={"ID":"971d1be6-8abd-4920-a123-724c9d972483","Type":"ContainerDied","Data":"9db11c82745dd8c50288a838bec32d4bfe15c088bd3d8731548382d96b9d89a3"} Nov 01 01:00:03.534858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.534840 4910 scope.go:122] "RemoveContainer" containerID="9db11c82745dd8c50288a838bec32d4bfe15c088bd3d8731548382d96b9d89a3" Nov 01 01:00:03.536773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.536751 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" event={"ID":"ebe5eb59-798f-4b82-b8fc-459178830ba8","Type":"ContainerStarted","Data":"5b7a805020208d020e9c9bfd2107d0f44b28c15e2879793fe27444e2a511aa05"} Nov 01 01:00:03.538542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.538519 4910 generic.go:356] "Generic (PLEG): container finished" podID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerID="81c924074fde81f98700fb9d8b33693466cf0479798f930bf1f86aaad5706a11" exitCode=60 Nov 01 01:00:03.538685 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.538601 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" event={"ID":"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6","Type":"ContainerDied","Data":"81c924074fde81f98700fb9d8b33693466cf0479798f930bf1f86aaad5706a11"} Nov 01 01:00:03.538873 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.538853 4910 scope.go:122] "RemoveContainer" containerID="81c924074fde81f98700fb9d8b33693466cf0479798f930bf1f86aaad5706a11" Nov 01 01:00:03.540772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.540749 4910 generic.go:356] "Generic (PLEG): container finished" podID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerID="1899af572444841bd96c65308b0839ab606001d6d5a4d27d5001fcd8769e4f98" exitCode=60 Nov 01 01:00:03.540874 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.540825 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" event={"ID":"1e090764-767d-4f29-bd16-a94cee6b5d82","Type":"ContainerDied","Data":"1899af572444841bd96c65308b0839ab606001d6d5a4d27d5001fcd8769e4f98"} Nov 01 01:00:03.543805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.543785 4910 scope.go:122] "RemoveContainer" containerID="1899af572444841bd96c65308b0839ab606001d6d5a4d27d5001fcd8769e4f98" Nov 01 01:00:03.544115 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.544094 4910 generic.go:356] "Generic (PLEG): container finished" podID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerID="fee4f18fbee778e7c6f2e5aec0dd07b867dd7d02f67e0a6f05ed849964e04cad" exitCode=60 Nov 01 01:00:03.544339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.544230 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" event={"ID":"d8617a04-4c0e-4ea6-ae6c-75925a6af130","Type":"ContainerDied","Data":"fee4f18fbee778e7c6f2e5aec0dd07b867dd7d02f67e0a6f05ed849964e04cad"} Nov 01 01:00:03.544410 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.544383 4910 scope.go:122] "RemoveContainer" containerID="fee4f18fbee778e7c6f2e5aec0dd07b867dd7d02f67e0a6f05ed849964e04cad" Nov 01 01:00:03.546112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.546070 4910 generic.go:356] "Generic (PLEG): container finished" podID="231d97be-9f94-4289-aafe-32330f14ceb2" containerID="23513d68362dc68b988516497199c4955afe65030c9cfb97233d6565408d7e2c" exitCode=60 Nov 01 01:00:03.546210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.546120 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerDied","Data":"23513d68362dc68b988516497199c4955afe65030c9cfb97233d6565408d7e2c"} Nov 01 01:00:03.546390 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.546375 4910 scope.go:122] "RemoveContainer" containerID="23513d68362dc68b988516497199c4955afe65030c9cfb97233d6565408d7e2c" Nov 01 01:00:03.547769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.547748 4910 generic.go:356] "Generic (PLEG): container finished" podID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerID="7e24a4d2024ca638f21efd4dcee255ef038bba453c92bf0affcb2f29fd0dc5c8" exitCode=60 Nov 01 01:00:03.547879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.547816 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" event={"ID":"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a","Type":"ContainerDied","Data":"7e24a4d2024ca638f21efd4dcee255ef038bba453c92bf0affcb2f29fd0dc5c8"} Nov 01 01:00:03.547991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.547975 4910 scope.go:122] "RemoveContainer" containerID="7e24a4d2024ca638f21efd4dcee255ef038bba453c92bf0affcb2f29fd0dc5c8" Nov 01 01:00:03.684373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.680964 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" podStartSLOduration=3.6809450999999997 podStartE2EDuration="3.6809451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:03.559210628 +0000 UTC m=+2193.538792965" watchObservedRunningTime="2026-11-01 01:00:03.6809451 +0000 UTC m=+2193.660527435" Nov 01 01:00:03.910009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:03.909955 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" podStartSLOduration=3.909941967 podStartE2EDuration="3.909941967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:03.904732721 +0000 UTC m=+2193.884315057" watchObservedRunningTime="2026-11-01 01:00:03.909941967 +0000 UTC m=+2193.889524301" Nov 01 01:00:04.600581 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.599024 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" event={"ID":"971d1be6-8abd-4920-a123-724c9d972483","Type":"ContainerStarted","Data":"33d6dd29179851bd053ba6aea9493f4bf54d6179f77a556f417e8fa8f89d77e4"} Nov 01 01:00:04.612792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.611757 4910 generic.go:356] "Generic (PLEG): container finished" podID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerID="5b7a805020208d020e9c9bfd2107d0f44b28c15e2879793fe27444e2a511aa05" exitCode=60 Nov 01 01:00:04.612792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.611841 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" event={"ID":"ebe5eb59-798f-4b82-b8fc-459178830ba8","Type":"ContainerDied","Data":"5b7a805020208d020e9c9bfd2107d0f44b28c15e2879793fe27444e2a511aa05"} Nov 01 01:00:04.612792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.611891 4910 scope.go:122] "RemoveContainer" containerID="c07af8633d6ae5b5499f940baf51466ed0fe1d90a5015c3cd4d46b531e1e2579" Nov 01 01:00:04.612792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.612366 4910 scope.go:122] "RemoveContainer" containerID="5b7a805020208d020e9c9bfd2107d0f44b28c15e2879793fe27444e2a511aa05" Nov 01 01:00:04.614370 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:04.613833 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp_odh-ai-gateway-infra(ebe5eb59-798f-4b82-b8fc-459178830ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" Nov 01 01:00:04.616685 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.616661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" event={"ID":"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6","Type":"ContainerStarted","Data":"32149f2a8afa66bbcbf17062bc5c9d4ef517f50007ebbaaa9478ffecd7a85d3a"} Nov 01 01:00:04.619953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.619935 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" event={"ID":"1e090764-767d-4f29-bd16-a94cee6b5d82","Type":"ContainerStarted","Data":"90a35923510ffbeccba9a28a2537bf4790a0a4b23b0e43666ec3a5c4afc8f1ec"} Nov 01 01:00:04.622868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.622841 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" event={"ID":"d8617a04-4c0e-4ea6-ae6c-75925a6af130","Type":"ContainerStarted","Data":"8b93b7aa4dd44fc6a5a47eb68bcae288a380c796147521eab5419fcc02ef21ab"} Nov 01 01:00:04.625510 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.625493 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerStarted","Data":"96c049256d13c836dd3869dd1fcf8cad6925beaeb05c3b01bc4350ef4afa7192"} Nov 01 01:00:04.627524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.627505 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" event={"ID":"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a","Type":"ContainerStarted","Data":"c2c958cc48c8db4a581692432097529ab0ee09b51ba41eca70fc54f72f2c567c"} Nov 01 01:00:04.629962 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.629938 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerStarted","Data":"ea2a19ab7d3e53008195a9836c118aa0b603d1df28a4887c702601a850394425"} Nov 01 01:00:04.633009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:04.632965 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerStarted","Data":"f63117bcea901eb9f967b48115c8d07bf0324893463b6973fcd0695135fceca3"} Nov 01 01:00:05.020769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.020712 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" podStartSLOduration=5.020696866 podStartE2EDuration="5.020696866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:05.018810336 +0000 UTC m=+2194.998392699" watchObservedRunningTime="2026-11-01 01:00:05.020696866 +0000 UTC m=+2195.000279206" Nov 01 01:00:05.650089 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.650043 4910 generic.go:356] "Generic (PLEG): container finished" podID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerID="f63117bcea901eb9f967b48115c8d07bf0324893463b6973fcd0695135fceca3" exitCode=60 Nov 01 01:00:05.650770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.650210 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerDied","Data":"f63117bcea901eb9f967b48115c8d07bf0324893463b6973fcd0695135fceca3"} Nov 01 01:00:05.650770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.650312 4910 scope.go:122] "RemoveContainer" containerID="0810149aea1e16aa1b66b77a26571bb4b3dfb657839d3b9ad40b8c64d923778f" Nov 01 01:00:05.650770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.650547 4910 scope.go:122] "RemoveContainer" containerID="f63117bcea901eb9f967b48115c8d07bf0324893463b6973fcd0695135fceca3" Nov 01 01:00:05.651084 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.651042 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8_odh-ai-gateway-infra(4b4bf6f4-4336-4fa5-a8ed-adae92b6a626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" Nov 01 01:00:05.654576 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.654551 4910 generic.go:356] "Generic (PLEG): container finished" podID="8a75cd45-43a4-439c-aac7-638366038761" containerID="043f38c534e95c3fd47638cf9a0747e69d1a98a491e542562afebcf5b5f0619a" exitCode=60 Nov 01 01:00:05.654576 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.654652 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" event={"ID":"8a75cd45-43a4-439c-aac7-638366038761","Type":"ContainerDied","Data":"043f38c534e95c3fd47638cf9a0747e69d1a98a491e542562afebcf5b5f0619a"} Nov 01 01:00:05.655138 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.655120 4910 scope.go:122] "RemoveContainer" containerID="043f38c534e95c3fd47638cf9a0747e69d1a98a491e542562afebcf5b5f0619a" Nov 01 01:00:05.655565 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.655531 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr_odh-ai-gateway-infra(8a75cd45-43a4-439c-aac7-638366038761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" podUID="8a75cd45-43a4-439c-aac7-638366038761" Nov 01 01:00:05.656902 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.656883 4910 generic.go:356] "Generic (PLEG): container finished" podID="971d1be6-8abd-4920-a123-724c9d972483" containerID="33d6dd29179851bd053ba6aea9493f4bf54d6179f77a556f417e8fa8f89d77e4" exitCode=60 Nov 01 01:00:05.657043 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.657016 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" event={"ID":"971d1be6-8abd-4920-a123-724c9d972483","Type":"ContainerDied","Data":"33d6dd29179851bd053ba6aea9493f4bf54d6179f77a556f417e8fa8f89d77e4"} Nov 01 01:00:05.657304 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.657288 4910 scope.go:122] "RemoveContainer" containerID="33d6dd29179851bd053ba6aea9493f4bf54d6179f77a556f417e8fa8f89d77e4" Nov 01 01:00:05.657710 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.657688 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2_odh-ai-gateway-infra(971d1be6-8abd-4920-a123-724c9d972483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" podUID="971d1be6-8abd-4920-a123-724c9d972483" Nov 01 01:00:05.659441 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.659421 4910 scope.go:122] "RemoveContainer" containerID="5b7a805020208d020e9c9bfd2107d0f44b28c15e2879793fe27444e2a511aa05" Nov 01 01:00:05.659829 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.659794 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp_odh-ai-gateway-infra(ebe5eb59-798f-4b82-b8fc-459178830ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" Nov 01 01:00:05.661546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.661524 4910 generic.go:356] "Generic (PLEG): container finished" podID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerID="32149f2a8afa66bbcbf17062bc5c9d4ef517f50007ebbaaa9478ffecd7a85d3a" exitCode=60 Nov 01 01:00:05.661661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.661596 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" event={"ID":"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6","Type":"ContainerDied","Data":"32149f2a8afa66bbcbf17062bc5c9d4ef517f50007ebbaaa9478ffecd7a85d3a"} Nov 01 01:00:05.661975 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.661957 4910 scope.go:122] "RemoveContainer" containerID="32149f2a8afa66bbcbf17062bc5c9d4ef517f50007ebbaaa9478ffecd7a85d3a" Nov 01 01:00:05.662332 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.662307 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w_odh-ai-gateway-infra(d55cc5a9-f6b9-4055-af1d-ae6eca77ade6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" Nov 01 01:00:05.664005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.663983 4910 generic.go:356] "Generic (PLEG): container finished" podID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerID="90a35923510ffbeccba9a28a2537bf4790a0a4b23b0e43666ec3a5c4afc8f1ec" exitCode=60 Nov 01 01:00:05.664119 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.664072 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" event={"ID":"1e090764-767d-4f29-bd16-a94cee6b5d82","Type":"ContainerDied","Data":"90a35923510ffbeccba9a28a2537bf4790a0a4b23b0e43666ec3a5c4afc8f1ec"} Nov 01 01:00:05.664303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.664286 4910 scope.go:122] "RemoveContainer" containerID="90a35923510ffbeccba9a28a2537bf4790a0a4b23b0e43666ec3a5c4afc8f1ec" Nov 01 01:00:05.664593 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.664570 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx_odh-ai-gateway-infra(1e090764-767d-4f29-bd16-a94cee6b5d82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" Nov 01 01:00:05.666213 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.666194 4910 generic.go:356] "Generic (PLEG): container finished" podID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerID="8b93b7aa4dd44fc6a5a47eb68bcae288a380c796147521eab5419fcc02ef21ab" exitCode=60 Nov 01 01:00:05.666336 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.666279 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" event={"ID":"d8617a04-4c0e-4ea6-ae6c-75925a6af130","Type":"ContainerDied","Data":"8b93b7aa4dd44fc6a5a47eb68bcae288a380c796147521eab5419fcc02ef21ab"} Nov 01 01:00:05.666597 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.666464 4910 scope.go:122] "RemoveContainer" containerID="8b93b7aa4dd44fc6a5a47eb68bcae288a380c796147521eab5419fcc02ef21ab" Nov 01 01:00:05.666856 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.666814 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp_odh-ai-gateway-infra(d8617a04-4c0e-4ea6-ae6c-75925a6af130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" Nov 01 01:00:05.668462 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.668445 4910 scope.go:122] "RemoveContainer" containerID="898e15516d75948da7f46310b61d7bf107d65adc58f5f69949abbac491fff26c" Nov 01 01:00:05.668871 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.668850 4910 generic.go:356] "Generic (PLEG): container finished" podID="231d97be-9f94-4289-aafe-32330f14ceb2" containerID="96c049256d13c836dd3869dd1fcf8cad6925beaeb05c3b01bc4350ef4afa7192" exitCode=60 Nov 01 01:00:05.669932 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.668931 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerDied","Data":"96c049256d13c836dd3869dd1fcf8cad6925beaeb05c3b01bc4350ef4afa7192"} Nov 01 01:00:05.669932 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.669254 4910 scope.go:122] "RemoveContainer" containerID="96c049256d13c836dd3869dd1fcf8cad6925beaeb05c3b01bc4350ef4afa7192" Nov 01 01:00:05.669932 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.669493 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn_odh-ai-gateway-infra(231d97be-9f94-4289-aafe-32330f14ceb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" Nov 01 01:00:05.671814 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.671794 4910 generic.go:356] "Generic (PLEG): container finished" podID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerID="c2c958cc48c8db4a581692432097529ab0ee09b51ba41eca70fc54f72f2c567c" exitCode=60 Nov 01 01:00:05.671929 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.671910 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" event={"ID":"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a","Type":"ContainerDied","Data":"c2c958cc48c8db4a581692432097529ab0ee09b51ba41eca70fc54f72f2c567c"} Nov 01 01:00:05.672107 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.672087 4910 scope.go:122] "RemoveContainer" containerID="c2c958cc48c8db4a581692432097529ab0ee09b51ba41eca70fc54f72f2c567c" Nov 01 01:00:05.672389 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.672369 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr_odh-ai-gateway-infra(4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" Nov 01 01:00:05.674458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.674423 4910 generic.go:356] "Generic (PLEG): container finished" podID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerID="ea2a19ab7d3e53008195a9836c118aa0b603d1df28a4887c702601a850394425" exitCode=60 Nov 01 01:00:05.674548 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.674502 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerDied","Data":"ea2a19ab7d3e53008195a9836c118aa0b603d1df28a4887c702601a850394425"} Nov 01 01:00:05.674833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.674815 4910 scope.go:122] "RemoveContainer" containerID="ea2a19ab7d3e53008195a9836c118aa0b603d1df28a4887c702601a850394425" Nov 01 01:00:05.675096 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:05.675076 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25_odh-ai-gateway-infra(c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" Nov 01 01:00:05.686080 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.686044 4910 scope.go:122] "RemoveContainer" containerID="9db11c82745dd8c50288a838bec32d4bfe15c088bd3d8731548382d96b9d89a3" Nov 01 01:00:05.704669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.704647 4910 scope.go:122] "RemoveContainer" containerID="81c924074fde81f98700fb9d8b33693466cf0479798f930bf1f86aaad5706a11" Nov 01 01:00:05.720461 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.720438 4910 scope.go:122] "RemoveContainer" containerID="1899af572444841bd96c65308b0839ab606001d6d5a4d27d5001fcd8769e4f98" Nov 01 01:00:05.738611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.738586 4910 scope.go:122] "RemoveContainer" containerID="fee4f18fbee778e7c6f2e5aec0dd07b867dd7d02f67e0a6f05ed849964e04cad" Nov 01 01:00:05.761523 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.759028 4910 scope.go:122] "RemoveContainer" containerID="23513d68362dc68b988516497199c4955afe65030c9cfb97233d6565408d7e2c" Nov 01 01:00:05.776711 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.776686 4910 scope.go:122] "RemoveContainer" containerID="7e24a4d2024ca638f21efd4dcee255ef038bba453c92bf0affcb2f29fd0dc5c8" Nov 01 01:00:05.790497 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:05.790479 4910 scope.go:122] "RemoveContainer" containerID="d730dcaa3292ec05d1d9abd6bd5f997a55ed11ecf072fe2b237aaa33c8708371" Nov 01 01:00:06.696510 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.696474 4910 scope.go:122] "RemoveContainer" containerID="f63117bcea901eb9f967b48115c8d07bf0324893463b6973fcd0695135fceca3" Nov 01 01:00:06.697036 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.696941 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8_odh-ai-gateway-infra(4b4bf6f4-4336-4fa5-a8ed-adae92b6a626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" Nov 01 01:00:06.700652 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.700600 4910 scope.go:122] "RemoveContainer" containerID="33d6dd29179851bd053ba6aea9493f4bf54d6179f77a556f417e8fa8f89d77e4" Nov 01 01:00:06.700975 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.700954 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2_odh-ai-gateway-infra(971d1be6-8abd-4920-a123-724c9d972483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" podUID="971d1be6-8abd-4920-a123-724c9d972483" Nov 01 01:00:06.702508 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.702488 4910 scope.go:122] "RemoveContainer" containerID="32149f2a8afa66bbcbf17062bc5c9d4ef517f50007ebbaaa9478ffecd7a85d3a" Nov 01 01:00:06.702857 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.702836 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w_odh-ai-gateway-infra(d55cc5a9-f6b9-4055-af1d-ae6eca77ade6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" Nov 01 01:00:06.704369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.704350 4910 scope.go:122] "RemoveContainer" containerID="90a35923510ffbeccba9a28a2537bf4790a0a4b23b0e43666ec3a5c4afc8f1ec" Nov 01 01:00:06.704681 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.704656 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx_odh-ai-gateway-infra(1e090764-767d-4f29-bd16-a94cee6b5d82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" Nov 01 01:00:06.706092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.706074 4910 scope.go:122] "RemoveContainer" containerID="8b93b7aa4dd44fc6a5a47eb68bcae288a380c796147521eab5419fcc02ef21ab" Nov 01 01:00:06.706349 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.706328 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp_odh-ai-gateway-infra(d8617a04-4c0e-4ea6-ae6c-75925a6af130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" Nov 01 01:00:06.708029 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.708011 4910 scope.go:122] "RemoveContainer" containerID="96c049256d13c836dd3869dd1fcf8cad6925beaeb05c3b01bc4350ef4afa7192" Nov 01 01:00:06.708232 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.708204 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn_odh-ai-gateway-infra(231d97be-9f94-4289-aafe-32330f14ceb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" Nov 01 01:00:06.709905 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.709882 4910 scope.go:122] "RemoveContainer" containerID="c2c958cc48c8db4a581692432097529ab0ee09b51ba41eca70fc54f72f2c567c" Nov 01 01:00:06.710125 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.710099 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr_odh-ai-gateway-infra(4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" Nov 01 01:00:06.711579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:06.711561 4910 scope.go:122] "RemoveContainer" containerID="ea2a19ab7d3e53008195a9836c118aa0b603d1df28a4887c702601a850394425" Nov 01 01:00:06.711787 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:06.711764 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25_odh-ai-gateway-infra(c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" Nov 01 01:00:07.721578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:07.721541 4910 generic.go:356] "Generic (PLEG): container finished" podID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerID="4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018" exitCode=0 Nov 01 01:00:07.722060 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:07.721614 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" event={"ID":"51cf8936-fd4f-4915-acf0-c4cdabc12636","Type":"ContainerDied","Data":"4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018"} Nov 01 01:00:07.722105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:07.722095 4910 scope.go:122] "RemoveContainer" containerID="4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018" Nov 01 01:00:07.723941 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:07.723912 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:07.724027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:07.723982 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:07.724027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:07.724014 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:08.732669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:08.732597 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" event={"ID":"51cf8936-fd4f-4915-acf0-c4cdabc12636","Type":"ContainerStarted","Data":"b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93"} Nov 01 01:00:09.074683 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:09.060000 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 01:00:09.074683 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:09.060179 4910 patch_prober.go:29] interesting pod/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 01:00:09.074683 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:09.060227 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 01:00:09.823450 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:09.823419 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 01:00:10.066649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:10.059988 4910 patch_prober.go:29] interesting pod/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 01:00:10.066649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:10.060051 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 01:00:11.072650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:11.062740 4910 patch_prober.go:29] interesting pod/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 01:00:11.072650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:11.062807 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 01:00:12.057221 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.057189 4910 patch_prober.go:29] interesting pod/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 01:00:12.057500 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.057472 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 01:00:12.207642 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.205760 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd"] Nov 01 01:00:12.211045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.211027 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.284009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.283973 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0f9d81-openshift-default-dockercfg-442kp\"" Nov 01 01:00:12.381075 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.380961 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4749198e-b5c8-404f-88ca-f01bd68817a7-istiod-ca-cert\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.381410 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.381388 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-data\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.381638 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.381602 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcpl\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-kube-api-access-pdcpl\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.381817 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.381790 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-socket\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.381938 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.381925 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-podinfo\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.382123 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.382107 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-token\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.387373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.382222 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-certs\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.387647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.387604 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-envoy\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.387864 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.387848 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-credential-socket\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.388193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.388178 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd"] Nov 01 01:00:12.547497 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.547460 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4749198e-b5c8-404f-88ca-f01bd68817a7-istiod-ca-cert\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.559701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.548159 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-data\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.615610 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcpl\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-kube-api-access-pdcpl\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.615734 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-socket\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.615781 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-podinfo\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.615830 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-token\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.615864 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-certs\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.615941 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-envoy\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.616060 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-credential-socket\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.550263 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4749198e-b5c8-404f-88ca-f01bd68817a7-istiod-ca-cert\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.550705 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-data\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.617992 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-socket\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.618259 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-certs\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.623444 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-envoy\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.627041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.624318 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-podinfo\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.628048 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.628027 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-credential-socket\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.700365 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.700246 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-token\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.702373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.702346 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcpl\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-kube-api-access-pdcpl\") pod \"e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:12.829723 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:12.829685 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:13.058657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.057995 4910 patch_prober.go:29] interesting pod/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 01:00:13.058657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.058055 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 01:00:13.130934 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:13.130893 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:00:13.309027 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:00:13.308952 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4749198e_b5c8_404f_88ca_f01bd68817a7.slice/crio-1f86b2fd336d5f58bac3e4aff08185ba452170511bf9601c07073249882f8d11 WatchSource:0}: Error finding container 1f86b2fd336d5f58bac3e4aff08185ba452170511bf9601c07073249882f8d11: Status 404 returned error can't find the container with id 1f86b2fd336d5f58bac3e4aff08185ba452170511bf9601c07073249882f8d11 Nov 01 01:00:13.312417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.312379 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:13.312524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.312449 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:13.312524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.312490 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:13.316561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.316005 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd"] Nov 01 01:00:13.820105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.820063 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" event={"ID":"4749198e-b5c8-404f-88ca-f01bd68817a7","Type":"ContainerStarted","Data":"892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0"} Nov 01 01:00:13.820283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.820135 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" event={"ID":"4749198e-b5c8-404f-88ca-f01bd68817a7","Type":"ContainerStarted","Data":"1f86b2fd336d5f58bac3e4aff08185ba452170511bf9601c07073249882f8d11"} Nov 01 01:00:13.829840 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.829795 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:13.945154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:13.945102 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podStartSLOduration=1.945087412 podStartE2EDuration="1.945087412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:13.940585141 +0000 UTC m=+2203.920167476" watchObservedRunningTime="2026-11-01 01:00:13.945087412 +0000 UTC m=+2203.924669746" Nov 01 01:00:14.170485 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:14.170401 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 01:00:14.173058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:14.173035 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 01:00:14.836904 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:14.836874 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:15.835365 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:15.835334 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:15.836283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:15.836264 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:00:16.185175 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:16.185088 4910 scope.go:122] "RemoveContainer" containerID="5b7a805020208d020e9c9bfd2107d0f44b28c15e2879793fe27444e2a511aa05" Nov 01 01:00:16.846076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:16.845975 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" event={"ID":"ebe5eb59-798f-4b82-b8fc-459178830ba8","Type":"ContainerStarted","Data":"7b2ede51e17288905184ccc2889cdd43fe7c88ec28e13430bdb4ebfaaa53603f"} Nov 01 01:00:17.184677 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.184577 4910 scope.go:122] "RemoveContainer" containerID="043f38c534e95c3fd47638cf9a0747e69d1a98a491e542562afebcf5b5f0619a" Nov 01 01:00:17.184845 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.184718 4910 scope.go:122] "RemoveContainer" containerID="f63117bcea901eb9f967b48115c8d07bf0324893463b6973fcd0695135fceca3" Nov 01 01:00:17.184953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.184914 4910 scope.go:122] "RemoveContainer" containerID="96c049256d13c836dd3869dd1fcf8cad6925beaeb05c3b01bc4350ef4afa7192" Nov 01 01:00:17.185115 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.185094 4910 scope.go:122] "RemoveContainer" containerID="ea2a19ab7d3e53008195a9836c118aa0b603d1df28a4887c702601a850394425" Nov 01 01:00:17.860683 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.860655 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerStarted","Data":"3c4a7447ae46542d92a9c174f3f176652e1e02ce02a404bf5c4ce7b3f6adaaed"} Nov 01 01:00:17.863551 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.863522 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerStarted","Data":"43982b5416c6208a1c8edf200bd1b1981ec673ca5626596b1c52f3f8b850463b"} Nov 01 01:00:17.866043 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.866016 4910 generic.go:356] "Generic (PLEG): container finished" podID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerID="7b2ede51e17288905184ccc2889cdd43fe7c88ec28e13430bdb4ebfaaa53603f" exitCode=60 Nov 01 01:00:17.866183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.866164 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" event={"ID":"ebe5eb59-798f-4b82-b8fc-459178830ba8","Type":"ContainerDied","Data":"7b2ede51e17288905184ccc2889cdd43fe7c88ec28e13430bdb4ebfaaa53603f"} Nov 01 01:00:17.866304 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.866217 4910 scope.go:122] "RemoveContainer" containerID="5b7a805020208d020e9c9bfd2107d0f44b28c15e2879793fe27444e2a511aa05" Nov 01 01:00:17.866698 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.866677 4910 scope.go:122] "RemoveContainer" containerID="7b2ede51e17288905184ccc2889cdd43fe7c88ec28e13430bdb4ebfaaa53603f" Nov 01 01:00:17.867041 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:17.867018 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp_odh-ai-gateway-infra(ebe5eb59-798f-4b82-b8fc-459178830ba8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" Nov 01 01:00:17.871195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.871167 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerStarted","Data":"c288e7fa54bfe4c2bb1552c54b20265b10fb3308f14edbf9cf5f79733a97ee83"} Nov 01 01:00:17.985914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:17.985879 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp"] Nov 01 01:00:18.185439 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.185347 4910 scope.go:122] "RemoveContainer" containerID="90a35923510ffbeccba9a28a2537bf4790a0a4b23b0e43666ec3a5c4afc8f1ec" Nov 01 01:00:18.185663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.185604 4910 scope.go:122] "RemoveContainer" containerID="8b93b7aa4dd44fc6a5a47eb68bcae288a380c796147521eab5419fcc02ef21ab" Nov 01 01:00:18.185816 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.185783 4910 scope.go:122] "RemoveContainer" containerID="33d6dd29179851bd053ba6aea9493f4bf54d6179f77a556f417e8fa8f89d77e4" Nov 01 01:00:18.185965 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.185914 4910 scope.go:122] "RemoveContainer" containerID="c2c958cc48c8db4a581692432097529ab0ee09b51ba41eca70fc54f72f2c567c" Nov 01 01:00:18.186374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.186352 4910 scope.go:122] "RemoveContainer" containerID="32149f2a8afa66bbcbf17062bc5c9d4ef517f50007ebbaaa9478ffecd7a85d3a" Nov 01 01:00:18.891061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.890889 4910 generic.go:356] "Generic (PLEG): container finished" podID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerID="3c4a7447ae46542d92a9c174f3f176652e1e02ce02a404bf5c4ce7b3f6adaaed" exitCode=60 Nov 01 01:00:18.891061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.890950 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerDied","Data":"3c4a7447ae46542d92a9c174f3f176652e1e02ce02a404bf5c4ce7b3f6adaaed"} Nov 01 01:00:18.891061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.891023 4910 scope.go:122] "RemoveContainer" containerID="ea2a19ab7d3e53008195a9836c118aa0b603d1df28a4887c702601a850394425" Nov 01 01:00:18.891588 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.891419 4910 scope.go:122] "RemoveContainer" containerID="3c4a7447ae46542d92a9c174f3f176652e1e02ce02a404bf5c4ce7b3f6adaaed" Nov 01 01:00:18.891971 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:18.891907 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25_odh-ai-gateway-infra(c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" Nov 01 01:00:18.894785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.894757 4910 generic.go:356] "Generic (PLEG): container finished" podID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerID="43982b5416c6208a1c8edf200bd1b1981ec673ca5626596b1c52f3f8b850463b" exitCode=60 Nov 01 01:00:18.894880 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.894801 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerDied","Data":"43982b5416c6208a1c8edf200bd1b1981ec673ca5626596b1c52f3f8b850463b"} Nov 01 01:00:18.895288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.895271 4910 scope.go:122] "RemoveContainer" containerID="43982b5416c6208a1c8edf200bd1b1981ec673ca5626596b1c52f3f8b850463b" Nov 01 01:00:18.895682 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:18.895658 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8_odh-ai-gateway-infra(4b4bf6f4-4336-4fa5-a8ed-adae92b6a626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" Nov 01 01:00:18.897164 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.897140 4910 generic.go:356] "Generic (PLEG): container finished" podID="8a75cd45-43a4-439c-aac7-638366038761" containerID="71c07945a67a98e687334c7d52545d9f36d288d04f19fed725954876b997d6fd" exitCode=60 Nov 01 01:00:18.897263 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.897211 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" event={"ID":"8a75cd45-43a4-439c-aac7-638366038761","Type":"ContainerDied","Data":"71c07945a67a98e687334c7d52545d9f36d288d04f19fed725954876b997d6fd"} Nov 01 01:00:18.897683 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.897664 4910 scope.go:122] "RemoveContainer" containerID="71c07945a67a98e687334c7d52545d9f36d288d04f19fed725954876b997d6fd" Nov 01 01:00:18.898039 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:18.898014 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr_odh-ai-gateway-infra(8a75cd45-43a4-439c-aac7-638366038761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" podUID="8a75cd45-43a4-439c-aac7-638366038761" Nov 01 01:00:18.901677 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.901654 4910 generic.go:356] "Generic (PLEG): container finished" podID="231d97be-9f94-4289-aafe-32330f14ceb2" containerID="c288e7fa54bfe4c2bb1552c54b20265b10fb3308f14edbf9cf5f79733a97ee83" exitCode=60 Nov 01 01:00:18.901780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.901708 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerDied","Data":"c288e7fa54bfe4c2bb1552c54b20265b10fb3308f14edbf9cf5f79733a97ee83"} Nov 01 01:00:18.902053 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.902036 4910 scope.go:122] "RemoveContainer" containerID="c288e7fa54bfe4c2bb1552c54b20265b10fb3308f14edbf9cf5f79733a97ee83" Nov 01 01:00:18.902345 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:18.902325 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn_odh-ai-gateway-infra(231d97be-9f94-4289-aafe-32330f14ceb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" Nov 01 01:00:18.954465 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.954445 4910 scope.go:122] "RemoveContainer" containerID="f63117bcea901eb9f967b48115c8d07bf0324893463b6973fcd0695135fceca3" Nov 01 01:00:18.982811 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.982544 4910 scope.go:122] "RemoveContainer" containerID="043f38c534e95c3fd47638cf9a0747e69d1a98a491e542562afebcf5b5f0619a" Nov 01 01:00:18.992924 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:18.992899 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25"] Nov 01 01:00:19.012252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.011875 4910 scope.go:122] "RemoveContainer" containerID="96c049256d13c836dd3869dd1fcf8cad6925beaeb05c3b01bc4350ef4afa7192" Nov 01 01:00:19.097611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.097587 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:19.107671 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.107603 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8"] Nov 01 01:00:19.148681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.145541 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe5eb59-798f-4b82-b8fc-459178830ba8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe5eb59-798f-4b82-b8fc-459178830ba8-maas-api-service-ca\") pod \"ebe5eb59-798f-4b82-b8fc-459178830ba8\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " Nov 01 01:00:19.148681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.145728 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe5eb59-798f-4b82-b8fc-459178830ba8-kube-api-access-2qdzf\" (UniqueName: \"kubernetes.io/projected/ebe5eb59-798f-4b82-b8fc-459178830ba8-kube-api-access-2qdzf\") pod \"ebe5eb59-798f-4b82-b8fc-459178830ba8\" (UID: \"ebe5eb59-798f-4b82-b8fc-459178830ba8\") " Nov 01 01:00:19.156767 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.156736 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe5eb59-798f-4b82-b8fc-459178830ba8-maas-api-service-ca" pod "ebe5eb59-798f-4b82-b8fc-459178830ba8" (UID: "ebe5eb59-798f-4b82-b8fc-459178830ba8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:19.166083 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.166052 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe5eb59-798f-4b82-b8fc-459178830ba8-kube-api-access-2qdzf" pod "ebe5eb59-798f-4b82-b8fc-459178830ba8" (UID: "ebe5eb59-798f-4b82-b8fc-459178830ba8"). InnerVolumeSpecName "kube-api-access-2qdzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:19.248040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.248008 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdzf\" (UniqueName: \"kubernetes.io/projected/ebe5eb59-798f-4b82-b8fc-459178830ba8-kube-api-access-2qdzf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:19.248154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.248045 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe5eb59-798f-4b82-b8fc-459178830ba8-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:19.402133 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.402043 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn"] Nov 01 01:00:19.914618 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.914557 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" Nov 01 01:00:19.914618 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.914551 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp" event={"ID":"ebe5eb59-798f-4b82-b8fc-459178830ba8","Type":"ContainerDied","Data":"afa7b4b790f2623a5749e36418b172d3733fa272cb85e36a8e22694c1fec4306"} Nov 01 01:00:19.914618 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.914711 4910 scope.go:122] "RemoveContainer" containerID="7b2ede51e17288905184ccc2889cdd43fe7c88ec28e13430bdb4ebfaaa53603f" Nov 01 01:00:19.917301 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.917274 4910 generic.go:356] "Generic (PLEG): container finished" podID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerID="6b0b8e6c8a2bc6093edba860736d32bce66aa96d8a8d478e89a2a8d92739c0d3" exitCode=60 Nov 01 01:00:19.917402 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.917338 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" event={"ID":"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6","Type":"ContainerDied","Data":"6b0b8e6c8a2bc6093edba860736d32bce66aa96d8a8d478e89a2a8d92739c0d3"} Nov 01 01:00:19.917870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.917850 4910 scope.go:122] "RemoveContainer" containerID="6b0b8e6c8a2bc6093edba860736d32bce66aa96d8a8d478e89a2a8d92739c0d3" Nov 01 01:00:19.918325 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:19.918274 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w_odh-ai-gateway-infra(d55cc5a9-f6b9-4055-af1d-ae6eca77ade6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" Nov 01 01:00:19.919544 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.919518 4910 generic.go:356] "Generic (PLEG): container finished" podID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerID="d6be2f7342e8dcb889276cd348f361abe62037d35040d45a9f3003b54666c9ec" exitCode=60 Nov 01 01:00:19.919668 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.919585 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" event={"ID":"1e090764-767d-4f29-bd16-a94cee6b5d82","Type":"ContainerDied","Data":"d6be2f7342e8dcb889276cd348f361abe62037d35040d45a9f3003b54666c9ec"} Nov 01 01:00:19.920052 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.920034 4910 scope.go:122] "RemoveContainer" containerID="d6be2f7342e8dcb889276cd348f361abe62037d35040d45a9f3003b54666c9ec" Nov 01 01:00:19.920560 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:19.920536 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx_odh-ai-gateway-infra(1e090764-767d-4f29-bd16-a94cee6b5d82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" Nov 01 01:00:19.921591 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.921573 4910 generic.go:356] "Generic (PLEG): container finished" podID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerID="47b3db889efbed5529f66f88b021ef837848dd4139e487ff5f9728f6383e2c5f" exitCode=60 Nov 01 01:00:19.921706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.921649 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" event={"ID":"d8617a04-4c0e-4ea6-ae6c-75925a6af130","Type":"ContainerDied","Data":"47b3db889efbed5529f66f88b021ef837848dd4139e487ff5f9728f6383e2c5f"} Nov 01 01:00:19.922100 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.922052 4910 scope.go:122] "RemoveContainer" containerID="47b3db889efbed5529f66f88b021ef837848dd4139e487ff5f9728f6383e2c5f" Nov 01 01:00:19.922464 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:19.922422 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp_odh-ai-gateway-infra(d8617a04-4c0e-4ea6-ae6c-75925a6af130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" Nov 01 01:00:19.926176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.926155 4910 generic.go:356] "Generic (PLEG): container finished" podID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerID="a44f24a29cb05dc842cb2ed7548e156e8cae7dbc3e22c4532a54128821218bcd" exitCode=60 Nov 01 01:00:19.926261 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.926224 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" event={"ID":"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a","Type":"ContainerDied","Data":"a44f24a29cb05dc842cb2ed7548e156e8cae7dbc3e22c4532a54128821218bcd"} Nov 01 01:00:19.926571 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.926555 4910 scope.go:122] "RemoveContainer" containerID="a44f24a29cb05dc842cb2ed7548e156e8cae7dbc3e22c4532a54128821218bcd" Nov 01 01:00:19.926886 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:19.926862 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr_odh-ai-gateway-infra(4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" Nov 01 01:00:19.927807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.927747 4910 scope.go:122] "RemoveContainer" containerID="32149f2a8afa66bbcbf17062bc5c9d4ef517f50007ebbaaa9478ffecd7a85d3a" Nov 01 01:00:19.935300 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.935277 4910 generic.go:356] "Generic (PLEG): container finished" podID="971d1be6-8abd-4920-a123-724c9d972483" containerID="7830d7c34fa83ffe8f7d7949be379b992ca0915a92c4608997d0ac6ec38101fe" exitCode=60 Nov 01 01:00:19.935405 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.935329 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" event={"ID":"971d1be6-8abd-4920-a123-724c9d972483","Type":"ContainerDied","Data":"7830d7c34fa83ffe8f7d7949be379b992ca0915a92c4608997d0ac6ec38101fe"} Nov 01 01:00:19.935769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.935746 4910 scope.go:122] "RemoveContainer" containerID="7830d7c34fa83ffe8f7d7949be379b992ca0915a92c4608997d0ac6ec38101fe" Nov 01 01:00:19.936114 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:19.936090 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2_odh-ai-gateway-infra(971d1be6-8abd-4920-a123-724c9d972483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" podUID="971d1be6-8abd-4920-a123-724c9d972483" Nov 01 01:00:19.945871 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.945476 4910 scope.go:122] "RemoveContainer" containerID="90a35923510ffbeccba9a28a2537bf4790a0a4b23b0e43666ec3a5c4afc8f1ec" Nov 01 01:00:19.976330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.975869 4910 scope.go:122] "RemoveContainer" containerID="8b93b7aa4dd44fc6a5a47eb68bcae288a380c796147521eab5419fcc02ef21ab" Nov 01 01:00:19.998320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:19.998294 4910 scope.go:122] "RemoveContainer" containerID="c2c958cc48c8db4a581692432097529ab0ee09b51ba41eca70fc54f72f2c567c" Nov 01 01:00:20.025418 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.025177 4910 scope.go:122] "RemoveContainer" containerID="33d6dd29179851bd053ba6aea9493f4bf54d6179f77a556f417e8fa8f89d77e4" Nov 01 01:00:20.156422 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.156323 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:20.158097 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.158062 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr"] Nov 01 01:00:20.184052 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.183970 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-maas-api-service-ca\") pod \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " Nov 01 01:00:20.185068 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.184311 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-kube-api-access-sqll2\" (UniqueName: \"kubernetes.io/projected/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-kube-api-access-sqll2\") pod \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\" (UID: \"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626\") " Nov 01 01:00:20.185068 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.184556 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-maas-api-service-ca" pod "4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" (UID: "4b4bf6f4-4336-4fa5-a8ed-adae92b6a626"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:20.185068 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.185032 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.201397 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.200772 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-kube-api-access-sqll2" pod "4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" (UID: "4b4bf6f4-4336-4fa5-a8ed-adae92b6a626"). InnerVolumeSpecName "kube-api-access-sqll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:20.216872 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.216782 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:20.222394 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.222372 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:20.290176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.285890 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-maas-api-service-ca\") pod \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " Nov 01 01:00:20.290176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.285975 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231d97be-9f94-4289-aafe-32330f14ceb2-kube-api-access-49sdj\" (UniqueName: \"kubernetes.io/projected/231d97be-9f94-4289-aafe-32330f14ceb2-kube-api-access-49sdj\") pod \"231d97be-9f94-4289-aafe-32330f14ceb2\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " Nov 01 01:00:20.290176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.286045 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-kube-api-access-25x8d\" (UniqueName: \"kubernetes.io/projected/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-kube-api-access-25x8d\") pod \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\" (UID: \"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4\") " Nov 01 01:00:20.290176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.286086 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231d97be-9f94-4289-aafe-32330f14ceb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d97be-9f94-4289-aafe-32330f14ceb2-maas-api-service-ca\") pod \"231d97be-9f94-4289-aafe-32330f14ceb2\" (UID: \"231d97be-9f94-4289-aafe-32330f14ceb2\") " Nov 01 01:00:20.290176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.287074 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqll2\" (UniqueName: \"kubernetes.io/projected/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626-kube-api-access-sqll2\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.290176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.287856 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-maas-api-service-ca" pod "c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" (UID: "c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:20.295390 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.295348 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231d97be-9f94-4289-aafe-32330f14ceb2-maas-api-service-ca" pod "231d97be-9f94-4289-aafe-32330f14ceb2" (UID: "231d97be-9f94-4289-aafe-32330f14ceb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:20.298590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.298555 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231d97be-9f94-4289-aafe-32330f14ceb2-kube-api-access-49sdj" pod "231d97be-9f94-4289-aafe-32330f14ceb2" (UID: "231d97be-9f94-4289-aafe-32330f14ceb2"). InnerVolumeSpecName "kube-api-access-49sdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:20.299333 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.299300 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-kube-api-access-25x8d" pod "c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" (UID: "c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4"). InnerVolumeSpecName "kube-api-access-25x8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:20.388530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.388499 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.388672 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.388536 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49sdj\" (UniqueName: \"kubernetes.io/projected/231d97be-9f94-4289-aafe-32330f14ceb2-kube-api-access-49sdj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.388672 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.388552 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25x8d\" (UniqueName: \"kubernetes.io/projected/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4-kube-api-access-25x8d\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.388672 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.388564 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d97be-9f94-4289-aafe-32330f14ceb2-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.407506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.407475 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:20.489372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.489276 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a75cd45-43a4-439c-aac7-638366038761-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a75cd45-43a4-439c-aac7-638366038761-maas-api-service-ca\") pod \"8a75cd45-43a4-439c-aac7-638366038761\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " Nov 01 01:00:20.489537 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.489511 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a75cd45-43a4-439c-aac7-638366038761-kube-api-access-mw2dz\" (UniqueName: \"kubernetes.io/projected/8a75cd45-43a4-439c-aac7-638366038761-kube-api-access-mw2dz\") pod \"8a75cd45-43a4-439c-aac7-638366038761\" (UID: \"8a75cd45-43a4-439c-aac7-638366038761\") " Nov 01 01:00:20.489708 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.489678 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a75cd45-43a4-439c-aac7-638366038761-maas-api-service-ca" pod "8a75cd45-43a4-439c-aac7-638366038761" (UID: "8a75cd45-43a4-439c-aac7-638366038761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:20.490185 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.490167 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a75cd45-43a4-439c-aac7-638366038761-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.491974 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.491951 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a75cd45-43a4-439c-aac7-638366038761-kube-api-access-mw2dz" pod "8a75cd45-43a4-439c-aac7-638366038761" (UID: "8a75cd45-43a4-439c-aac7-638366038761"). InnerVolumeSpecName "kube-api-access-mw2dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:20.591292 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.591255 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw2dz\" (UniqueName: \"kubernetes.io/projected/8a75cd45-43a4-439c-aac7-638366038761-kube-api-access-mw2dz\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:20.674059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.674026 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp"] Nov 01 01:00:20.679755 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.679724 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af82f1-29891580-97cnp"] Nov 01 01:00:20.952907 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.952872 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" event={"ID":"231d97be-9f94-4289-aafe-32330f14ceb2","Type":"ContainerDied","Data":"4220d861edc3b31a6af1e3147bb28076af22b799e32b04f4d6d80ef6917027f2"} Nov 01 01:00:20.953343 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.952882 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn" Nov 01 01:00:20.953343 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.952964 4910 scope.go:122] "RemoveContainer" containerID="c288e7fa54bfe4c2bb1552c54b20265b10fb3308f14edbf9cf5f79733a97ee83" Nov 01 01:00:20.956220 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.956197 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" Nov 01 01:00:20.956328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.956197 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25" event={"ID":"c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4","Type":"ContainerDied","Data":"12d2450d2b63b189d1217f618d256bf3709dd46f2b01287c203da9444e84587b"} Nov 01 01:00:20.957791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.957775 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" Nov 01 01:00:20.957919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.957839 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8" event={"ID":"4b4bf6f4-4336-4fa5-a8ed-adae92b6a626","Type":"ContainerDied","Data":"e61f834f3edd7e4bde8d875e0ea396077be1dcbd55350205291ec9aa1c1f8a2c"} Nov 01 01:00:20.960669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.959529 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" event={"ID":"8a75cd45-43a4-439c-aac7-638366038761","Type":"ContainerDied","Data":"59b24f0fc19e9ee0ab0fcb266afc7764fe09724097ec557c26c00fa9df8684f8"} Nov 01 01:00:20.960669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.959537 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr" Nov 01 01:00:20.971372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.971346 4910 scope.go:122] "RemoveContainer" containerID="3c4a7447ae46542d92a9c174f3f176652e1e02ce02a404bf5c4ce7b3f6adaaed" Nov 01 01:00:20.981731 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.981712 4910 scope.go:122] "RemoveContainer" containerID="43982b5416c6208a1c8edf200bd1b1981ec673ca5626596b1c52f3f8b850463b" Nov 01 01:00:20.991803 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:20.991751 4910 scope.go:122] "RemoveContainer" containerID="71c07945a67a98e687334c7d52545d9f36d288d04f19fed725954876b997d6fd" Nov 01 01:00:21.101129 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.101094 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2"] Nov 01 01:00:21.241084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.240767 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8"] Nov 01 01:00:21.254761 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.254722 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp"] Nov 01 01:00:21.294567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.294535 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ba5b10-29891580-k5wm8"] Nov 01 01:00:21.344514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.344477 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx"] Nov 01 01:00:21.540811 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.483317 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn"] Nov 01 01:00:21.540811 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.483374 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891580-jd8qn"] Nov 01 01:00:21.633240 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.575323 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w"] Nov 01 01:00:21.659049 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.659022 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:21.679706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.679675 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25"] Nov 01 01:00:21.687089 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.687059 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-30fb19-29891580-wgl25"] Nov 01 01:00:21.748650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.747597 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971d1be6-8abd-4920-a123-724c9d972483-kube-api-access-7thck\" (UniqueName: \"kubernetes.io/projected/971d1be6-8abd-4920-a123-724c9d972483-kube-api-access-7thck\") pod \"971d1be6-8abd-4920-a123-724c9d972483\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " Nov 01 01:00:21.748650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.747684 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971d1be6-8abd-4920-a123-724c9d972483-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971d1be6-8abd-4920-a123-724c9d972483-maas-api-service-ca\") pod \"971d1be6-8abd-4920-a123-724c9d972483\" (UID: \"971d1be6-8abd-4920-a123-724c9d972483\") " Nov 01 01:00:21.750372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.750325 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971d1be6-8abd-4920-a123-724c9d972483-maas-api-service-ca" pod "971d1be6-8abd-4920-a123-724c9d972483" (UID: "971d1be6-8abd-4920-a123-724c9d972483"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:21.755804 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.755769 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971d1be6-8abd-4920-a123-724c9d972483-kube-api-access-7thck" pod "971d1be6-8abd-4920-a123-724c9d972483" (UID: "971d1be6-8abd-4920-a123-724c9d972483"). InnerVolumeSpecName "kube-api-access-7thck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:21.802737 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.801830 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:21.928652 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.925824 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thck\" (UniqueName: \"kubernetes.io/projected/971d1be6-8abd-4920-a123-724c9d972483-kube-api-access-7thck\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:21.928652 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:21.925859 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971d1be6-8abd-4920-a123-724c9d972483-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.071714 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.031550 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8617a04-4c0e-4ea6-ae6c-75925a6af130-kube-api-access-4q7s4\" (UniqueName: \"kubernetes.io/projected/d8617a04-4c0e-4ea6-ae6c-75925a6af130-kube-api-access-4q7s4\") pod \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " Nov 01 01:00:22.071714 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.031610 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8617a04-4c0e-4ea6-ae6c-75925a6af130-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8617a04-4c0e-4ea6-ae6c-75925a6af130-maas-api-service-ca\") pod \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\" (UID: \"d8617a04-4c0e-4ea6-ae6c-75925a6af130\") " Nov 01 01:00:22.071714 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.069828 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr"] Nov 01 01:00:22.080420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.080354 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-99eb0f-29891580-gqjzr"] Nov 01 01:00:22.081388 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.081058 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" event={"ID":"d8617a04-4c0e-4ea6-ae6c-75925a6af130","Type":"ContainerDied","Data":"23452b198f68781f95e5a711d2449756654a194f19d9d5ebb055ce1cc161ae1f"} Nov 01 01:00:22.081388 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.081118 4910 scope.go:122] "RemoveContainer" containerID="47b3db889efbed5529f66f88b021ef837848dd4139e487ff5f9728f6383e2c5f" Nov 01 01:00:22.081388 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.081208 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp" Nov 01 01:00:22.090656 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.087932 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8617a04-4c0e-4ea6-ae6c-75925a6af130-maas-api-service-ca" pod "d8617a04-4c0e-4ea6-ae6c-75925a6af130" (UID: "d8617a04-4c0e-4ea6-ae6c-75925a6af130"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:22.107126 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.099183 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8617a04-4c0e-4ea6-ae6c-75925a6af130-kube-api-access-4q7s4" pod "d8617a04-4c0e-4ea6-ae6c-75925a6af130" (UID: "d8617a04-4c0e-4ea6-ae6c-75925a6af130"). InnerVolumeSpecName "kube-api-access-4q7s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:22.127704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.127427 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" event={"ID":"971d1be6-8abd-4920-a123-724c9d972483","Type":"ContainerDied","Data":"e323ca72ce44ea013b2c548bf21c00cda85f2b43c84558844ab6b540edad8e60"} Nov 01 01:00:22.127704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.127557 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2" Nov 01 01:00:22.151918 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.147849 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q7s4\" (UniqueName: \"kubernetes.io/projected/d8617a04-4c0e-4ea6-ae6c-75925a6af130-kube-api-access-4q7s4\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.151918 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.147884 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8617a04-4c0e-4ea6-ae6c-75925a6af130-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.316955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.316926 4910 scope.go:122] "RemoveContainer" containerID="7830d7c34fa83ffe8f7d7949be379b992ca0915a92c4608997d0ac6ec38101fe" Nov 01 01:00:22.328256 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.327521 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" path="/var/lib/kubelet/pods/231d97be-9f94-4289-aafe-32330f14ceb2/volumes" Nov 01 01:00:22.328256 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.328234 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" path="/var/lib/kubelet/pods/4b4bf6f4-4336-4fa5-a8ed-adae92b6a626/volumes" Nov 01 01:00:22.328873 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.328852 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a75cd45-43a4-439c-aac7-638366038761" path="/var/lib/kubelet/pods/8a75cd45-43a4-439c-aac7-638366038761/volumes" Nov 01 01:00:22.330282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.330260 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" path="/var/lib/kubelet/pods/c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4/volumes" Nov 01 01:00:22.331025 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.331005 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" path="/var/lib/kubelet/pods/ebe5eb59-798f-4b82-b8fc-459178830ba8/volumes" Nov 01 01:00:22.350214 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.350171 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc"] Nov 01 01:00:22.363877 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.363846 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:22.368258 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.368214 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:00:22.373246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.373222 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:00:22.373585 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.373566 4910 cpu_manager.go:414] "containerMap: removing container" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:00:22.376259 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.376237 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:00:22.379719 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379697 4910 cpu_manager.go:414] "containerMap: removing container" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.379830 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379740 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.379830 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379771 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:00:22.379830 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379779 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379830 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379840 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379856 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379863 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379887 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379895 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379951 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379960 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379977 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379984 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.379995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.379995 4910 cpu_manager.go:414] "containerMap: removing container" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380002 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380067 4910 cpu_manager.go:414] "containerMap: removing container" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380077 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380093 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380100 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380122 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380130 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380154 4910 cpu_manager.go:414] "containerMap: removing container" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380162 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380182 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380189 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380213 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380222 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380236 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380244 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380288 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380297 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380328 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:00:22.380550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380336 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380753 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380781 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380791 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380802 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380826 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380845 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380858 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380867 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380885 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380895 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380912 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380923 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380931 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380951 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380968 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.380988 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.381002 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.381020 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.381031 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4bf6f4-4336-4fa5-a8ed-adae92b6a626" containerName="cleanup" Nov 01 01:00:22.382252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.381057 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:00:22.388649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.388167 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e090764-767d-4f29-bd16-a94cee6b5d82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e090764-767d-4f29-bd16-a94cee6b5d82-maas-api-service-ca\") pod \"1e090764-767d-4f29-bd16-a94cee6b5d82\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " Nov 01 01:00:22.388649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.388287 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e090764-767d-4f29-bd16-a94cee6b5d82-kube-api-access-wcsnw\" (UniqueName: \"kubernetes.io/projected/1e090764-767d-4f29-bd16-a94cee6b5d82-kube-api-access-wcsnw\") pod \"1e090764-767d-4f29-bd16-a94cee6b5d82\" (UID: \"1e090764-767d-4f29-bd16-a94cee6b5d82\") " Nov 01 01:00:22.388843 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.388818 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e090764-767d-4f29-bd16-a94cee6b5d82-maas-api-service-ca" pod "1e090764-767d-4f29-bd16-a94cee6b5d82" (UID: "1e090764-767d-4f29-bd16-a94cee6b5d82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:22.389420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.389397 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e090764-767d-4f29-bd16-a94cee6b5d82-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.402744 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.402420 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e090764-767d-4f29-bd16-a94cee6b5d82-kube-api-access-wcsnw" pod "1e090764-767d-4f29-bd16-a94cee6b5d82" (UID: "1e090764-767d-4f29-bd16-a94cee6b5d82"). InnerVolumeSpecName "kube-api-access-wcsnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:22.406396 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.406377 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr"] Nov 01 01:00:22.406536 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.406525 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc"] Nov 01 01:00:22.406929 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.406911 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.457846 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.457818 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:22.460164 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.460077 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-de2a95-openshift-default-dockercfg-pjd4r\"" Nov 01 01:00:22.490162 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.490127 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-maas-api-service-ca\") pod \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " Nov 01 01:00:22.491017 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.490993 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-kube-api-access-wgk9f\" (UniqueName: \"kubernetes.io/projected/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-kube-api-access-wgk9f\") pod \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\" (UID: \"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6\") " Nov 01 01:00:22.491550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.490707 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-maas-api-service-ca" pod "d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" (UID: "d55cc5a9-f6b9-4055-af1d-ae6eca77ade6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:22.492048 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.492024 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-credential-socket\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.492298 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.492257 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-certs\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.492500 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.492485 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-envoy\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.492870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.492851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istiod-ca-cert\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.493058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.493042 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-podinfo\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.493236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.493219 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-token\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.493398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.493384 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8s5q\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-kube-api-access-l8s5q\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.493555 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.493541 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-data\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.493723 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.493708 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-socket\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.493972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.493956 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.494070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.494058 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsnw\" (UniqueName: \"kubernetes.io/projected/1e090764-767d-4f29-bd16-a94cee6b5d82-kube-api-access-wcsnw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.495109 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.495087 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-kube-api-access-wgk9f" pod "d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" (UID: "d55cc5a9-f6b9-4055-af1d-ae6eca77ade6"). InnerVolumeSpecName "kube-api-access-wgk9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595490 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-credential-socket\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595559 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-certs\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595603 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-envoy\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595712 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istiod-ca-cert\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595748 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-podinfo\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595771 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-token\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595810 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8s5q\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-kube-api-access-l8s5q\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595839 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-data\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595864 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-socket\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.595957 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgk9f\" (UniqueName: \"kubernetes.io/projected/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6-kube-api-access-wgk9f\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.597208 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-certs\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.597412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-data\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.597588 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-credential-socket\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.598484 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istiod-ca-cert\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.598702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.598663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-socket\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.611479 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.611412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-podinfo\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.615581 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.615527 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-envoy\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.636017 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.635959 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp"] Nov 01 01:00:22.642940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.642903 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5a9c19-29891580-2dkfp"] Nov 01 01:00:22.692421 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.692280 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-token\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.693911 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.693850 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8s5q\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-kube-api-access-l8s5q\") pod \"e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.722405 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.722377 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:22.773986 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.773957 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:22.798914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.798878 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-kube-api-access-mmndb\" (UniqueName: \"kubernetes.io/projected/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-kube-api-access-mmndb\") pod \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " Nov 01 01:00:22.799038 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.799023 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-maas-api-service-ca\") pod \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\" (UID: \"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a\") " Nov 01 01:00:22.799365 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.799341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-maas-api-service-ca" pod "4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" (UID: "4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:00:22.799613 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.799595 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:22.801358 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.801334 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-kube-api-access-mmndb" pod "4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" (UID: "4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a"). InnerVolumeSpecName "kube-api-access-mmndb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:00:22.901611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:22.901577 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmndb\" (UniqueName: \"kubernetes.io/projected/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a-kube-api-access-mmndb\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:00:23.139187 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.139103 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" event={"ID":"d55cc5a9-f6b9-4055-af1d-ae6eca77ade6","Type":"ContainerDied","Data":"87a216f160fc5c60156ef4d4d2c5584dee501d473b835b498ef8a15079f29bbb"} Nov 01 01:00:23.139187 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.139122 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w" Nov 01 01:00:23.139187 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.139166 4910 scope.go:122] "RemoveContainer" containerID="6b0b8e6c8a2bc6093edba860736d32bce66aa96d8a8d478e89a2a8d92739c0d3" Nov 01 01:00:23.141840 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.141819 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" Nov 01 01:00:23.141840 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.141828 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx" event={"ID":"1e090764-767d-4f29-bd16-a94cee6b5d82","Type":"ContainerDied","Data":"4409e5084e6ad00d195cb329fb8fd9399089ddabaa0dbc607f9afa8368f865eb"} Nov 01 01:00:23.144557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.144535 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" Nov 01 01:00:23.144675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.144607 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr" event={"ID":"4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a","Type":"ContainerDied","Data":"d3073c54f871bdb851a304ba2d6a12b2883bafcc0a0da2f57d919a89c60aa041"} Nov 01 01:00:23.160099 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.160080 4910 scope.go:122] "RemoveContainer" containerID="d6be2f7342e8dcb889276cd348f361abe62037d35040d45a9f3003b54666c9ec" Nov 01 01:00:23.173363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.173336 4910 scope.go:122] "RemoveContainer" containerID="a44f24a29cb05dc842cb2ed7548e156e8cae7dbc3e22c4532a54128821218bcd" Nov 01 01:00:23.226387 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:00:23.226355 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c237a6_a5ed_488c_85d5_fa0ea4993c65.slice/crio-9564e37f46bb9886bd891e53953711def87f0ac67d312bd1fdd2763fbad86562 WatchSource:0}: Error finding container 9564e37f46bb9886bd891e53953711def87f0ac67d312bd1fdd2763fbad86562: Status 404 returned error can't find the container with id 9564e37f46bb9886bd891e53953711def87f0ac67d312bd1fdd2763fbad86562 Nov 01 01:00:23.228300 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.228279 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc"] Nov 01 01:00:23.229759 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.229720 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:23.229849 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.229831 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:23.229895 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.229878 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:23.240035 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.240009 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2"] Nov 01 01:00:23.248473 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.248445 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-27dee6-29891580-dnhj2"] Nov 01 01:00:23.469282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.469208 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr"] Nov 01 01:00:23.481296 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.481269 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-44c9bf-29891580-vzrwr"] Nov 01 01:00:23.622107 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.622076 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx"] Nov 01 01:00:23.628913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.628862 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-948e22-29891580-n8ftx"] Nov 01 01:00:23.752764 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.750667 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w"] Nov 01 01:00:23.775607 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:23.775567 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4143b7-29891580-bt59w"] Nov 01 01:00:24.158927 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.158835 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" event={"ID":"a3c237a6-a5ed-488c-85d5-fa0ea4993c65","Type":"ContainerStarted","Data":"1fc1fe192823f61332071797fc0848d3933f2f3324f7c856e5b7cfa506a78730"} Nov 01 01:00:24.158927 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.158896 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" event={"ID":"a3c237a6-a5ed-488c-85d5-fa0ea4993c65","Type":"ContainerStarted","Data":"9564e37f46bb9886bd891e53953711def87f0ac67d312bd1fdd2763fbad86562"} Nov 01 01:00:24.196839 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.196811 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" path="/var/lib/kubelet/pods/1e090764-767d-4f29-bd16-a94cee6b5d82/volumes" Nov 01 01:00:24.197286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.197265 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" path="/var/lib/kubelet/pods/4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a/volumes" Nov 01 01:00:24.197699 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.197684 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971d1be6-8abd-4920-a123-724c9d972483" path="/var/lib/kubelet/pods/971d1be6-8abd-4920-a123-724c9d972483/volumes" Nov 01 01:00:24.198567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.198547 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" path="/var/lib/kubelet/pods/d55cc5a9-f6b9-4055-af1d-ae6eca77ade6/volumes" Nov 01 01:00:24.199121 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.199102 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" path="/var/lib/kubelet/pods/d8617a04-4c0e-4ea6-ae6c-75925a6af130/volumes" Nov 01 01:00:24.284425 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.284378 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podStartSLOduration=3.284366525 podStartE2EDuration="3.284366525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:00:24.282465448 +0000 UTC m=+2214.262047799" watchObservedRunningTime="2026-11-01 01:00:24.284366525 +0000 UTC m=+2214.263948913" Nov 01 01:00:24.777805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.777743 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:24.785894 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:24.785854 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:25.171467 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:25.171385 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:25.172339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:25.172313 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:00:31.917934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:31.917904 4910 scope.go:122] "RemoveContainer" containerID="8e2794558d3e6b6626e1f75a119f0c2bd57c197750753723523871816705712c" Nov 01 01:00:43.105189 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:00:43.105148 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:00:56.606649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:56.599107 4910 generic.go:356] "Generic (PLEG): container finished" podID="266e279e-d94f-481c-af70-1c0eb92163d4" containerID="1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669" exitCode=0 Nov 01 01:00:56.606649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:56.599485 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" event={"ID":"266e279e-d94f-481c-af70-1c0eb92163d4","Type":"ContainerDied","Data":"1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669"} Nov 01 01:00:56.606649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:56.600156 4910 scope.go:122] "RemoveContainer" containerID="1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669" Nov 01 01:00:56.606649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:56.604948 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:56.606649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:56.605125 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:56.606649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:56.605222 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:00:57.627682 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:57.622817 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" event={"ID":"266e279e-d94f-481c-af70-1c0eb92163d4","Type":"ContainerStarted","Data":"a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9"} Nov 01 01:00:58.171657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:58.171032 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 01:00:58.171657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:58.171168 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 01:00:58.171657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:58.171230 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 01:00:59.172648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:59.169006 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 01:00:59.172648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:00:59.169091 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 01:01:00.170651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:00.166949 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 01:01:00.170651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:00.167007 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 01:01:01.091669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.088833 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw"] Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096097 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096133 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe5eb59-798f-4b82-b8fc-459178830ba8" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096151 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096160 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096191 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096199 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a75cd45-43a4-439c-aac7-638366038761" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096222 4910 cpu_manager.go:414] "containerMap: removing container" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096285 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096316 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096324 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e090764-767d-4f29-bd16-a94cee6b5d82" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096339 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096348 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096378 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096386 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096397 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096404 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096462 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096469 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096487 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096494 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096709 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096723 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096738 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b0d6a5-dbbf-4e24-a185-c24bb3a728b4" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096751 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096766 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="971d1be6-8abd-4920-a123-724c9d972483" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096776 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096786 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55cc5a9-f6b9-4055-af1d-ae6eca77ade6" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096796 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="231d97be-9f94-4289-aafe-32330f14ceb2" containerName="cleanup" Nov 01 01:01:01.097020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.096805 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8617a04-4c0e-4ea6-ae6c-75925a6af130" containerName="cleanup" Nov 01 01:01:01.117168 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.117135 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.167661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.166888 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 01:01:01.167661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.166963 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 01:01:01.169745 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.169715 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8fk\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-kube-api-access-ch8fk\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.175175 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.175140 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-data\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.175486 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.175466 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-podinfo\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.175724 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.175697 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-socket\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.175980 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.175961 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istiod-ca-cert\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.176160 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.176143 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-credential-socket\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.176377 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.176361 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-envoy\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.176571 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.176555 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-certs\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.176753 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.176735 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-token\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.218579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.218550 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f73fb0-openshift-default-dockercfg-v5mgx\"" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.396300 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw"] Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.407998 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-data\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408102 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-podinfo\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408139 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-socket\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408209 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istiod-ca-cert\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408246 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-credential-socket\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408323 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-envoy\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408382 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-certs\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408409 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-token\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.408465 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8fk\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-kube-api-access-ch8fk\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.409251 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-data\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.422893 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-podinfo\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.423206 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-socket\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.423840 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istiod-ca-cert\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.424104 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-credential-socket\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.444701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.433608 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-certs\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.516470 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.516403 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-token\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.612665 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.605167 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8fk\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-kube-api-access-ch8fk\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.636599 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.636557 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 01:01:01.717361 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.717266 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-envoy\") pod \"e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:01.858649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:01.854886 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:02.184716 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.173742 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 01:01:02.184716 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.173823 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 01:01:02.326861 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.326831 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw"] Nov 01 01:01:02.328881 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:01:02.328850 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b7ff3e_84d4_4241_81b0_5648a120a61a.slice/crio-e62ef410fdfddefa4ab8821e831039f5a183878a861127efa0ad0a08394c5add WatchSource:0}: Error finding container e62ef410fdfddefa4ab8821e831039f5a183878a861127efa0ad0a08394c5add: Status 404 returned error can't find the container with id e62ef410fdfddefa4ab8821e831039f5a183878a861127efa0ad0a08394c5add Nov 01 01:01:02.332822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.332783 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:02.332942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.332876 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:02.332942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.332921 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:02.729442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.729328 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" event={"ID":"c8b7ff3e-84d4-4241-81b0-5648a120a61a","Type":"ContainerStarted","Data":"8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8"} Nov 01 01:01:02.729442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.729400 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" event={"ID":"c8b7ff3e-84d4-4241-81b0-5648a120a61a","Type":"ContainerStarted","Data":"e62ef410fdfddefa4ab8821e831039f5a183878a861127efa0ad0a08394c5add"} Nov 01 01:01:02.864649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.864368 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:02.868704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:02.865535 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" podStartSLOduration=1.86551635 podStartE2EDuration="1.86551635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:02.854125483 +0000 UTC m=+2252.833707820" watchObservedRunningTime="2026-11-01 01:01:02.86551635 +0000 UTC m=+2252.845098686" Nov 01 01:01:03.168649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.168005 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 01:01:03.168649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.168065 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 01:01:03.239649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.237501 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7113-805306"] Nov 01 01:01:03.239649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.238720 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:03.239649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.238743 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:03.239649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.238939 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e80b1-4ed3-4b82-8e60-dfee9aa4c80a" containerName="cleanup" Nov 01 01:01:03.277968 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.276890 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7113-805306"] Nov 01 01:01:03.277968 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.277039 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:03.289256 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.289220 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmswn\" (UniqueName: \"kubernetes.io/projected/ceed49bd-049e-47cd-9513-3a56513400da-kube-api-access-wmswn\") pod \"test-curl-7113-805306\" (UID: \"ceed49bd-049e-47cd-9513-3a56513400da\") " pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:03.336267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.331706 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2k2hw\"" Nov 01 01:01:03.392498 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.392454 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmswn\" (UniqueName: \"kubernetes.io/projected/ceed49bd-049e-47cd-9513-3a56513400da-kube-api-access-wmswn\") pod \"test-curl-7113-805306\" (UID: \"ceed49bd-049e-47cd-9513-3a56513400da\") " pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:03.455826 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.455749 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmswn\" (UniqueName: \"kubernetes.io/projected/ceed49bd-049e-47cd-9513-3a56513400da-kube-api-access-wmswn\") pod \"test-curl-7113-805306\" (UID: \"ceed49bd-049e-47cd-9513-3a56513400da\") " pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:03.633289 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.633251 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:03.863180 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:03.863146 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:04.028679 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:01:04.028644 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceed49bd_049e_47cd_9513_3a56513400da.slice/crio-c2c94a961a3343225973b190a437db4f548669f548e66c6f3a5052290f6531c8 WatchSource:0}: Error finding container c2c94a961a3343225973b190a437db4f548669f548e66c6f3a5052290f6531c8: Status 404 returned error can't find the container with id c2c94a961a3343225973b190a437db4f548669f548e66c6f3a5052290f6531c8 Nov 01 01:01:04.029165 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.029144 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7113-805306"] Nov 01 01:01:04.170791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.170758 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 01:01:04.172267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.172237 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 01:01:04.753362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.753322 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7113-805306" event={"ID":"ceed49bd-049e-47cd-9513-3a56513400da","Type":"ContainerStarted","Data":"2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff"} Nov 01 01:01:04.753362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.753376 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7113-805306" event={"ID":"ceed49bd-049e-47cd-9513-3a56513400da","Type":"ContainerStarted","Data":"c2c94a961a3343225973b190a437db4f548669f548e66c6f3a5052290f6531c8"} Nov 01 01:01:04.754066 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.754038 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:04.754971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.754949 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:01:04.870140 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:04.870079 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7113-805306" podStartSLOduration=1.8700651449999999 podStartE2EDuration="1.870065145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:04.867922695 +0000 UTC m=+2254.847505031" watchObservedRunningTime="2026-11-01 01:01:04.870065145 +0000 UTC m=+2254.849647479" Nov 01 01:01:06.411385 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:06.411350 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7113-805306"] Nov 01 01:01:06.416218 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:06.416182 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7113-805306"] Nov 01 01:01:06.769673 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:06.769611 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7113-805306" podUID="ceed49bd-049e-47cd-9513-3a56513400da" containerName="test-curl-7113-805306" containerID="cri-o://2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff" gracePeriod=2 Nov 01 01:01:06.821560 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:01:06.821517 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-805306\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="ceed49bd-049e-47cd-9513-3a56513400da" pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:08.331712 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.331679 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx"] Nov 01 01:01:08.332567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.332543 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ceed49bd-049e-47cd-9513-3a56513400da" containerName="test-curl-7113-805306" Nov 01 01:01:08.332790 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.332568 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceed49bd-049e-47cd-9513-3a56513400da" containerName="test-curl-7113-805306" Nov 01 01:01:08.332790 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.332756 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceed49bd-049e-47cd-9513-3a56513400da" containerName="test-curl-7113-805306" Nov 01 01:01:08.335300 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.335280 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.372102 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.372069 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.372303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.372280 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.372421 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.372397 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqdv\" (UniqueName: \"kubernetes.io/projected/2de11323-a34f-4b75-80bf-4273146dbb90-kube-api-access-sfqdv\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.389130 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.389097 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:01:08.394478 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.394449 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx"] Nov 01 01:01:08.438569 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.438535 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:01:08.473596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.473563 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqdv\" (UniqueName: \"kubernetes.io/projected/2de11323-a34f-4b75-80bf-4273146dbb90-kube-api-access-sfqdv\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.473788 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.473705 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.473865 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.473838 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.474476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.474455 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.476347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.476306 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.536396 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.536355 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqdv\" (UniqueName: \"kubernetes.io/projected/2de11323-a34f-4b75-80bf-4273146dbb90-kube-api-access-sfqdv\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-xzcsx\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:08.648268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:08.648179 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:09.011414 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.011385 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:09.043853 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:01:09.043824 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de11323_a34f_4b75_80bf_4273146dbb90.slice/crio-6addf7058bf0fcb25fa1af321f89e8121a87854d0ac93f1236fb52a241548dd8 WatchSource:0}: Error finding container 6addf7058bf0fcb25fa1af321f89e8121a87854d0ac93f1236fb52a241548dd8: Status 404 returned error can't find the container with id 6addf7058bf0fcb25fa1af321f89e8121a87854d0ac93f1236fb52a241548dd8 Nov 01 01:01:09.044396 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.044270 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx"] Nov 01 01:01:09.088107 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.088081 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceed49bd-049e-47cd-9513-3a56513400da-kube-api-access-wmswn\" (UniqueName: \"kubernetes.io/projected/ceed49bd-049e-47cd-9513-3a56513400da-kube-api-access-wmswn\") pod \"ceed49bd-049e-47cd-9513-3a56513400da\" (UID: \"ceed49bd-049e-47cd-9513-3a56513400da\") " Nov 01 01:01:09.090671 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.090548 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceed49bd-049e-47cd-9513-3a56513400da-kube-api-access-wmswn" pod "ceed49bd-049e-47cd-9513-3a56513400da" (UID: "ceed49bd-049e-47cd-9513-3a56513400da"). InnerVolumeSpecName "kube-api-access-wmswn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:09.190005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.189912 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmswn\" (UniqueName: \"kubernetes.io/projected/ceed49bd-049e-47cd-9513-3a56513400da-kube-api-access-wmswn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:09.836649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.832244 4910 generic.go:356] "Generic (PLEG): container finished" podID="ceed49bd-049e-47cd-9513-3a56513400da" containerID="2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff" exitCode=137 Nov 01 01:01:09.836649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.832497 4910 scope.go:122] "RemoveContainer" containerID="2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff" Nov 01 01:01:09.836649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.832782 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:09.855678 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.854256 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" event={"ID":"2de11323-a34f-4b75-80bf-4273146dbb90","Type":"ContainerStarted","Data":"5565d103378f15d1d540043a0e9726d25dc11190ca4d68d9def6bafff0fc7dfe"} Nov 01 01:01:09.855678 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:09.854328 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" event={"ID":"2de11323-a34f-4b75-80bf-4273146dbb90","Type":"ContainerStarted","Data":"6addf7058bf0fcb25fa1af321f89e8121a87854d0ac93f1236fb52a241548dd8"} Nov 01 01:01:10.012875 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:10.012808 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" podStartSLOduration=2.01278839 podStartE2EDuration="2.01278839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:10.00991943 +0000 UTC m=+2259.989501766" watchObservedRunningTime="2026-11-01 01:01:10.01278839 +0000 UTC m=+2259.992370726" Nov 01 01:01:10.099771 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:01:10.093887 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-805306\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="ceed49bd-049e-47cd-9513-3a56513400da" pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:10.179052 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:10.179026 4910 scope.go:122] "RemoveContainer" containerID="2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff" Nov 01 01:01:10.179988 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:01:10.179961 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff\": container with ID starting with 2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff not found: ID does not exist" containerID="2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff" Nov 01 01:01:10.180167 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:10.180142 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff"} err="failed to get container status \"2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff\": rpc error: code = NotFound desc = could not find container \"2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff\": container with ID starting with 2a905958776fc0ece428a0a0696fcb199ae356d123333a0294d6961eba0671ff not found: ID does not exist" Nov 01 01:01:10.210038 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:10.205863 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceed49bd-049e-47cd-9513-3a56513400da" path="/var/lib/kubelet/pods/ceed49bd-049e-47cd-9513-3a56513400da/volumes" Nov 01 01:01:10.249499 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:01:10.249452 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7113-805306\" is forbidden: User \"system:node:ip-10-0-137-150.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-150.us-west-1.compute.internal' and this object" podUID="ceed49bd-049e-47cd-9513-3a56513400da" pod="opendatahub/test-curl-7113-805306" Nov 01 01:01:10.866293 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:10.866256 4910 generic.go:356] "Generic (PLEG): container finished" podID="2de11323-a34f-4b75-80bf-4273146dbb90" containerID="5565d103378f15d1d540043a0e9726d25dc11190ca4d68d9def6bafff0fc7dfe" exitCode=0 Nov 01 01:01:10.866793 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:10.866348 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" event={"ID":"2de11323-a34f-4b75-80bf-4273146dbb90","Type":"ContainerDied","Data":"5565d103378f15d1d540043a0e9726d25dc11190ca4d68d9def6bafff0fc7dfe"} Nov 01 01:01:12.088595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.088562 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:12.215229 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.215149 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-xks-ca\") pod \"2de11323-a34f-4b75-80bf-4273146dbb90\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " Nov 01 01:01:12.215396 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.215244 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-openshift-ca\") pod \"2de11323-a34f-4b75-80bf-4273146dbb90\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " Nov 01 01:01:12.215396 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.215381 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de11323-a34f-4b75-80bf-4273146dbb90-kube-api-access-sfqdv\" (UniqueName: \"kubernetes.io/projected/2de11323-a34f-4b75-80bf-4273146dbb90-kube-api-access-sfqdv\") pod \"2de11323-a34f-4b75-80bf-4273146dbb90\" (UID: \"2de11323-a34f-4b75-80bf-4273146dbb90\") " Nov 01 01:01:12.216154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.216122 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-openshift-ca" pod "2de11323-a34f-4b75-80bf-4273146dbb90" (UID: "2de11323-a34f-4b75-80bf-4273146dbb90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:01:12.217056 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.217025 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:12.225579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.219686 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-xks-ca" pod "2de11323-a34f-4b75-80bf-4273146dbb90" (UID: "2de11323-a34f-4b75-80bf-4273146dbb90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:01:12.235267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.235240 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de11323-a34f-4b75-80bf-4273146dbb90-kube-api-access-sfqdv" pod "2de11323-a34f-4b75-80bf-4273146dbb90" (UID: "2de11323-a34f-4b75-80bf-4273146dbb90"). InnerVolumeSpecName "kube-api-access-sfqdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:01:12.318223 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.318183 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de11323-a34f-4b75-80bf-4273146dbb90-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:12.318223 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.318216 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfqdv\" (UniqueName: \"kubernetes.io/projected/2de11323-a34f-4b75-80bf-4273146dbb90-kube-api-access-sfqdv\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:01:12.883461 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.883434 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" Nov 01 01:01:12.883660 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.883464 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx" event={"ID":"2de11323-a34f-4b75-80bf-4273146dbb90","Type":"ContainerDied","Data":"6addf7058bf0fcb25fa1af321f89e8121a87854d0ac93f1236fb52a241548dd8"} Nov 01 01:01:12.883660 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:12.883542 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6addf7058bf0fcb25fa1af321f89e8121a87854d0ac93f1236fb52a241548dd8" Nov 01 01:01:13.107273 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:01:13.107240 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:01:25.051736 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:25.051697 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh"] Nov 01 01:01:25.101354 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:25.101320 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd71bc49-bxnbh"] Nov 01 01:01:26.194299 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:26.194268 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb56c835-1d5e-48fa-944e-4adee75df049" path="/var/lib/kubelet/pods/fb56c835-1d5e-48fa-944e-4adee75df049/volumes" Nov 01 01:01:40.167635 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:40.167597 4910 generic.go:356] "Generic (PLEG): container finished" podID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerID="4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a" exitCode=0 Nov 01 01:01:40.168128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:40.167676 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" event={"ID":"44489bec-b295-4631-ae02-50ed5f97a2cd","Type":"ContainerDied","Data":"4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a"} Nov 01 01:01:40.168128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:40.168117 4910 scope.go:122] "RemoveContainer" containerID="4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a" Nov 01 01:01:40.169979 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:40.169946 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:40.170092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:40.170035 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:40.170163 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:40.170087 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:41.185988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:41.185958 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" event={"ID":"44489bec-b295-4631-ae02-50ed5f97a2cd","Type":"ContainerStarted","Data":"790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406"} Nov 01 01:01:41.896350 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:41.896300 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 01:01:42.105108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:42.105072 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 01:01:42.106472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:42.106449 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 01:01:42.106584 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:42.106504 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 01:01:43.105002 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:43.104967 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 01:01:43.105520 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:43.105053 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 01:01:43.106759 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:01:43.106729 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:01:44.104712 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:44.104676 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 01:01:44.104910 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:44.104731 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 01:01:45.104964 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:45.104925 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 01:01:45.105394 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:45.104988 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 01:01:46.105070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:46.105030 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 01:01:46.105484 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:46.105083 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 01:01:47.308052 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.308016 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 01:01:47.340532 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.340501 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 01:01:47.753765 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.753732 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq"] Nov 01 01:01:47.754467 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.754438 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2de11323-a34f-4b75-80bf-4273146dbb90" containerName="revoke-keys" Nov 01 01:01:47.754467 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.754458 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de11323-a34f-4b75-80bf-4273146dbb90" containerName="revoke-keys" Nov 01 01:01:47.754708 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.754580 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de11323-a34f-4b75-80bf-4273146dbb90" containerName="revoke-keys" Nov 01 01:01:47.757368 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.757349 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.820741 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.815495 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-72acf4-openshift-default-dockercfg-wg9h2\"" Nov 01 01:01:47.820741 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.819020 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq"] Nov 01 01:01:47.923648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923188 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-envoy\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.923648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923297 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-certs\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.923648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923359 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-podinfo\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.923648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923407 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7hf\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-kube-api-access-hj7hf\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.923648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923553 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-token\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.923648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923589 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54137c0-9741-4bf6-8c56-a57244f6d7db-istiod-ca-cert\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.924209 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923691 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-data\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.924209 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923728 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-socket\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:47.924209 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:47.923761 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-credential-socket\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.025667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025514 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-podinfo\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.025667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025598 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7hf\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-kube-api-access-hj7hf\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.025937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025748 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-token\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.025937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025779 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54137c0-9741-4bf6-8c56-a57244f6d7db-istiod-ca-cert\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.025937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025844 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-data\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.025937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025870 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-socket\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.025937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025890 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-credential-socket\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.026219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025948 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-envoy\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.026219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.025973 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-certs\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.026381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.026359 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-certs\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.027014 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.026986 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-data\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.027253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.027231 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-socket\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.027347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.027334 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54137c0-9741-4bf6-8c56-a57244f6d7db-istiod-ca-cert\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.027493 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.027469 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-credential-socket\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.039026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.038990 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-envoy\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.040139 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.040113 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-podinfo\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.091603 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.091572 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-token\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.093601 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.093571 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7hf\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-kube-api-access-hj7hf\") pod \"e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.373442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.373344 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:48.830310 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.830281 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq"] Nov 01 01:01:48.835932 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.835880 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:48.836075 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.836020 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:48.836137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:48.836078 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:01:49.359137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:49.359096 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" event={"ID":"d54137c0-9741-4bf6-8c56-a57244f6d7db","Type":"ContainerStarted","Data":"e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579"} Nov 01 01:01:49.359364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:49.359167 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" event={"ID":"d54137c0-9741-4bf6-8c56-a57244f6d7db","Type":"ContainerStarted","Data":"238afc1edd17baa41db52400fc891575ec0473a85fba01eb2862bab1206782b2"} Nov 01 01:01:49.373814 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:49.373779 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:49.380265 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:49.380237 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:49.514247 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:49.514187 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podStartSLOduration=2.514172468 podStartE2EDuration="2.514172468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:01:49.479543617 +0000 UTC m=+2299.459125953" watchObservedRunningTime="2026-11-01 01:01:49.514172468 +0000 UTC m=+2299.493754803" Nov 01 01:01:50.371361 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:50.371313 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:50.372499 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:50.372474 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:01:56.447565 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:56.445747 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz"] Nov 01 01:01:56.447565 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:56.446816 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" containerID="cri-o://2bdba2b4f4c476c2217b65b4510a6c2094bd0c9e6d95be29921dea8dc002352d" gracePeriod=30 Nov 01 01:01:56.485696 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:56.485280 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf"] Nov 01 01:01:56.486229 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:56.486176 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" containerID="cri-o://17e056297ca12db1bee566bfef3fd31884b750750dc6c759e53f6d5728979d34" gracePeriod=30 Nov 01 01:01:57.181183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:57.181150 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm"] Nov 01 01:01:57.181361 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:57.181343 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" containerID="cri-o://0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc" gracePeriod=30 Nov 01 01:01:59.978024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:59.977988 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:01:59.978424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:01:59.978038 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:02:01.575727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.575692 4910 generic.go:356] "Generic (PLEG): container finished" podID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerID="2bdba2b4f4c476c2217b65b4510a6c2094bd0c9e6d95be29921dea8dc002352d" exitCode=0 Nov 01 01:02:01.576114 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.575763 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" event={"ID":"ad371433-6a03-4a7b-abe1-21705b9a2ffb","Type":"ContainerDied","Data":"2bdba2b4f4c476c2217b65b4510a6c2094bd0c9e6d95be29921dea8dc002352d"} Nov 01 01:02:01.577800 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.577768 4910 generic.go:356] "Generic (PLEG): container finished" podID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerID="17e056297ca12db1bee566bfef3fd31884b750750dc6c759e53f6d5728979d34" exitCode=0 Nov 01 01:02:01.577911 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.577848 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" event={"ID":"8794b0b6-eadf-4bd6-ade4-eff3517b6552","Type":"ContainerDied","Data":"17e056297ca12db1bee566bfef3fd31884b750750dc6c759e53f6d5728979d34"} Nov 01 01:02:01.790707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.790677 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 01:02:01.830049 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.830020 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 01:02:01.838665 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.838613 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-certs\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.838822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.838683 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-kube-api-access-ld7xr\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-kube-api-access-ld7xr\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.838822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.838710 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istiod-ca-cert\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.838953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.838852 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-certs" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.838953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.838885 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-data\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.838953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.838939 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-socket\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.839125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.838981 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istiod-ca-cert\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839012 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-podinfo\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839066 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-data\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.839125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839114 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-token\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-token\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.839342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839163 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-data" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.839342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839189 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-podinfo\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.839342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839194 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istiod-ca-cert" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:01.839342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839232 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-credential-socket\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.839566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839370 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-envoy\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839395 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-data" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.839566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839428 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-certs\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839463 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-kube-api-access-rxx69\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-kube-api-access-rxx69\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839497 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-token\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-token\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839539 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-socket\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839952 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839580 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-credential-socket\") pod \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\" (UID: \"8794b0b6-eadf-4bd6-ade4-eff3517b6552\") " Nov 01 01:02:01.839952 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839605 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-envoy\") pod \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\" (UID: \"ad371433-6a03-4a7b-abe1-21705b9a2ffb\") " Nov 01 01:02:01.839952 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.839753 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istiod-ca-cert" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:01.840424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.840397 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.840528 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.840428 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.840528 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.840443 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.840528 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.840457 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.840528 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.840471 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.843434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.842111 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-socket" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.843434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.842352 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-credential-socket" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.843434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.842586 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-socket" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.843434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.842960 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-podinfo" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:02:01.843434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.843197 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-credential-socket" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.843855 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.843839 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-certs" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.844032 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.844007 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-kube-api-access-rxx69" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "kube-api-access-rxx69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:01.844105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.844026 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-envoy" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.844345 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.844316 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-token" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:01.844345 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.844328 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-podinfo" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:02:01.844607 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.844574 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-kube-api-access-ld7xr" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "kube-api-access-ld7xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:01.844827 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.844802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-envoy" pod "ad371433-6a03-4a7b-abe1-21705b9a2ffb" (UID: "ad371433-6a03-4a7b-abe1-21705b9a2ffb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:01.845154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.845134 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-token" pod "8794b0b6-eadf-4bd6-ade4-eff3517b6552" (UID: "8794b0b6-eadf-4bd6-ade4-eff3517b6552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:01.941463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941423 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld7xr\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-kube-api-access-ld7xr\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941460 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941470 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941479 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941489 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941498 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941507 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941515 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941523 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxx69\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-kube-api-access-rxx69\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941530 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8794b0b6-eadf-4bd6-ade4-eff3517b6552-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941539 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941546 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8794b0b6-eadf-4bd6-ade4-eff3517b6552-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:01.941743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:01.941557 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad371433-6a03-4a7b-abe1-21705b9a2ffb-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.566716 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.566686 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 01:02:02.567520 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567482 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-socket\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.567520 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567523 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-data\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.567729 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567555 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-certs\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.567729 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567594 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-envoy\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.567729 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567675 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-podinfo\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.567885 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567755 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-kube-api-access-xmlmd\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-kube-api-access-xmlmd\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.567885 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567786 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-credential-socket\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.567885 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567855 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-socket" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:02.568087 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.567913 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istiod-ca-cert\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.568087 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568069 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-token\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-token\") pod \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\" (UID: \"508f14e8-3bef-46c0-a6cc-55a430bb9a48\") " Nov 01 01:02:02.568312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568290 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-certs" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:02.568374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568308 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-credential-socket" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:02.568374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568345 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-data" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:02.568561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568537 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istiod-ca-cert" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:02.568920 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568901 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.569011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568926 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.569011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568940 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.569011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568954 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.569011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.568967 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.571676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.570988 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-podinfo" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:02:02.571676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.571025 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-envoy" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:02.571676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.571339 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-kube-api-access-xmlmd" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "kube-api-access-xmlmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:02.572712 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.572684 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-token" pod "508f14e8-3bef-46c0-a6cc-55a430bb9a48" (UID: "508f14e8-3bef-46c0-a6cc-55a430bb9a48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:02.603099 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.603038 4910 generic.go:356] "Generic (PLEG): container finished" podID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerID="0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc" exitCode=0 Nov 01 01:02:02.603533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.603160 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" event={"ID":"508f14e8-3bef-46c0-a6cc-55a430bb9a48","Type":"ContainerDied","Data":"0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc"} Nov 01 01:02:02.603533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.603217 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" Nov 01 01:02:02.603533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.603257 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm" event={"ID":"508f14e8-3bef-46c0-a6cc-55a430bb9a48","Type":"ContainerDied","Data":"8aadf25487a127d5753c74a81eb60a6836c4c153028ee723930ccd4656a273b8"} Nov 01 01:02:02.603533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.603301 4910 scope.go:122] "RemoveContainer" containerID="0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc" Nov 01 01:02:02.605842 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.605820 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" Nov 01 01:02:02.605941 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.605840 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz" event={"ID":"ad371433-6a03-4a7b-abe1-21705b9a2ffb","Type":"ContainerDied","Data":"7f932c4593159c11e79f16b38301e54f43b30eae54deaf3c4a5bb4fc3418bb36"} Nov 01 01:02:02.612092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.612061 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" event={"ID":"8794b0b6-eadf-4bd6-ade4-eff3517b6552","Type":"ContainerDied","Data":"66f4a49857227ee6a8993628c8bd60820065c6be5d57aa7af1b3d2da45a732e7"} Nov 01 01:02:02.612236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.612174 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf" Nov 01 01:02:02.622985 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.622965 4910 scope.go:122] "RemoveContainer" containerID="0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc" Nov 01 01:02:02.623307 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:02:02.623284 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc\": container with ID starting with 0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc not found: ID does not exist" containerID="0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc" Nov 01 01:02:02.623386 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.623322 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc"} err="failed to get container status \"0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc\": rpc error: code = NotFound desc = could not find container \"0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc\": container with ID starting with 0e1806aa711d027a53e9b9d63b2a34f2ec9a6857c2296fa91e1b1a613674f9dc not found: ID does not exist" Nov 01 01:02:02.623386 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.623346 4910 scope.go:122] "RemoveContainer" containerID="2bdba2b4f4c476c2217b65b4510a6c2094bd0c9e6d95be29921dea8dc002352d" Nov 01 01:02:02.633862 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.633837 4910 scope.go:122] "RemoveContainer" containerID="17e056297ca12db1bee566bfef3fd31884b750750dc6c759e53f6d5728979d34" Nov 01 01:02:02.670052 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.670022 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.670052 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.670048 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.670271 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.670063 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmlmd\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-kube-api-access-xmlmd\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.670271 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.670078 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f14e8-3bef-46c0-a6cc-55a430bb9a48-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:02.757195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757101 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289"] Nov 01 01:02:02.757766 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757748 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" Nov 01 01:02:02.757766 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757765 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" Nov 01 01:02:02.757958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757779 4910 cpu_manager.go:414] "containerMap: removing container" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" Nov 01 01:02:02.757958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757784 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" Nov 01 01:02:02.757958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757802 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" Nov 01 01:02:02.757958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757807 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" Nov 01 01:02:02.757958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757925 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" containerName="istio-proxy" Nov 01 01:02:02.757958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757936 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" containerName="istio-proxy" Nov 01 01:02:02.757958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.757945 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" containerName="istio-proxy" Nov 01 01:02:02.762911 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.762890 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.771455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.771427 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.771560 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.771545 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.771675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.771657 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvt7x\" (UniqueName: \"kubernetes.io/projected/20d4bab5-2798-4958-9538-b7fd86f5f30c-kube-api-access-mvt7x\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.815838 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.815809 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:02:02.816223 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.816195 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:02:02.818167 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.818142 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz"] Nov 01 01:02:02.825521 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.825453 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-tn8fz"] Nov 01 01:02:02.829725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.829608 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289"] Nov 01 01:02:02.873051 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.873017 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.873204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.873102 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.873290 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.873211 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvt7x\" (UniqueName: \"kubernetes.io/projected/20d4bab5-2798-4958-9538-b7fd86f5f30c-kube-api-access-mvt7x\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.874044 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.874015 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.875758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.875736 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:02.922813 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.922774 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf"] Nov 01 01:02:02.926949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.926929 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-wxrnf"] Nov 01 01:02:02.934580 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:02.934554 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvt7x\" (UniqueName: \"kubernetes.io/projected/20d4bab5-2798-4958-9538-b7fd86f5f30c-kube-api-access-mvt7x\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:03.046106 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:03.046026 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm"] Nov 01 01:02:03.049898 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:03.049876 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-92dcm"] Nov 01 01:02:03.076905 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:03.076873 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:03.487161 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:03.485304 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289"] Nov 01 01:02:03.489336 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:02:03.489305 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d4bab5_2798_4958_9538_b7fd86f5f30c.slice/crio-73b221522af938075a8dda34431109f834741218e692e68a23626347e9d11aa6 WatchSource:0}: Error finding container 73b221522af938075a8dda34431109f834741218e692e68a23626347e9d11aa6: Status 404 returned error can't find the container with id 73b221522af938075a8dda34431109f834741218e692e68a23626347e9d11aa6 Nov 01 01:02:03.622723 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:03.622687 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" event={"ID":"20d4bab5-2798-4958-9538-b7fd86f5f30c","Type":"ContainerStarted","Data":"73b221522af938075a8dda34431109f834741218e692e68a23626347e9d11aa6"} Nov 01 01:02:04.196374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:04.196298 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508f14e8-3bef-46c0-a6cc-55a430bb9a48" path="/var/lib/kubelet/pods/508f14e8-3bef-46c0-a6cc-55a430bb9a48/volumes" Nov 01 01:02:04.196824 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:04.196807 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8794b0b6-eadf-4bd6-ade4-eff3517b6552" path="/var/lib/kubelet/pods/8794b0b6-eadf-4bd6-ade4-eff3517b6552/volumes" Nov 01 01:02:04.197256 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:04.197242 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad371433-6a03-4a7b-abe1-21705b9a2ffb" path="/var/lib/kubelet/pods/ad371433-6a03-4a7b-abe1-21705b9a2ffb/volumes" Nov 01 01:02:04.654584 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:04.653755 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" event={"ID":"20d4bab5-2798-4958-9538-b7fd86f5f30c","Type":"ContainerStarted","Data":"a73d514a69a68a4aeaee27b610d17694ec1e69090db206832a65ffab1ccc49c0"} Nov 01 01:02:04.773106 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:04.773039 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" podStartSLOduration=2.773020031 podStartE2EDuration="2.773020031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:02:04.770830896 +0000 UTC m=+2314.750413232" watchObservedRunningTime="2026-11-01 01:02:04.773020031 +0000 UTC m=+2314.752602368" Nov 01 01:02:05.664094 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:05.664060 4910 generic.go:356] "Generic (PLEG): container finished" podID="20d4bab5-2798-4958-9538-b7fd86f5f30c" containerID="a73d514a69a68a4aeaee27b610d17694ec1e69090db206832a65ffab1ccc49c0" exitCode=0 Nov 01 01:02:05.664574 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:05.664144 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" event={"ID":"20d4bab5-2798-4958-9538-b7fd86f5f30c","Type":"ContainerDied","Data":"a73d514a69a68a4aeaee27b610d17694ec1e69090db206832a65ffab1ccc49c0"} Nov 01 01:02:06.837324 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.837298 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:06.931221 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.931157 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-openshift-ca\") pod \"20d4bab5-2798-4958-9538-b7fd86f5f30c\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " Nov 01 01:02:06.931348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.931259 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d4bab5-2798-4958-9538-b7fd86f5f30c-kube-api-access-mvt7x\" (UniqueName: \"kubernetes.io/projected/20d4bab5-2798-4958-9538-b7fd86f5f30c-kube-api-access-mvt7x\") pod \"20d4bab5-2798-4958-9538-b7fd86f5f30c\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " Nov 01 01:02:06.931348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.931328 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-xks-ca\") pod \"20d4bab5-2798-4958-9538-b7fd86f5f30c\" (UID: \"20d4bab5-2798-4958-9538-b7fd86f5f30c\") " Nov 01 01:02:06.931522 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.931501 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-openshift-ca" pod "20d4bab5-2798-4958-9538-b7fd86f5f30c" (UID: "20d4bab5-2798-4958-9538-b7fd86f5f30c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:06.931925 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.931891 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:06.933541 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.933510 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-xks-ca" pod "20d4bab5-2798-4958-9538-b7fd86f5f30c" (UID: "20d4bab5-2798-4958-9538-b7fd86f5f30c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:02:06.933689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:06.933672 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d4bab5-2798-4958-9538-b7fd86f5f30c-kube-api-access-mvt7x" pod "20d4bab5-2798-4958-9538-b7fd86f5f30c" (UID: "20d4bab5-2798-4958-9538-b7fd86f5f30c"). InnerVolumeSpecName "kube-api-access-mvt7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:07.033177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:07.033148 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvt7x\" (UniqueName: \"kubernetes.io/projected/20d4bab5-2798-4958-9538-b7fd86f5f30c-kube-api-access-mvt7x\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:07.033177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:07.033176 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d4bab5-2798-4958-9538-b7fd86f5f30c-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:07.681480 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:07.681453 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" Nov 01 01:02:07.681654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:07.681485 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289" event={"ID":"20d4bab5-2798-4958-9538-b7fd86f5f30c","Type":"ContainerDied","Data":"73b221522af938075a8dda34431109f834741218e692e68a23626347e9d11aa6"} Nov 01 01:02:07.681654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:07.681531 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b221522af938075a8dda34431109f834741218e692e68a23626347e9d11aa6" Nov 01 01:02:13.105064 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:02:13.105024 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:02:13.165054 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.165012 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7"] Nov 01 01:02:13.165792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.165771 4910 cpu_manager.go:414] "containerMap: removing container" podUID="20d4bab5-2798-4958-9538-b7fd86f5f30c" containerName="revoke-keys" Nov 01 01:02:13.165792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.165793 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d4bab5-2798-4958-9538-b7fd86f5f30c" containerName="revoke-keys" Nov 01 01:02:13.165999 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.165973 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d4bab5-2798-4958-9538-b7fd86f5f30c" containerName="revoke-keys" Nov 01 01:02:13.169742 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.169719 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.224432 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.224395 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.224594 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.224576 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.224762 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.224731 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km8tl\" (UniqueName: \"kubernetes.io/projected/94e95860-9cd7-4ced-bd39-a38242ff802b-kube-api-access-km8tl\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.226595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.226573 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7"] Nov 01 01:02:13.271076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.271035 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:02:13.271314 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.271141 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:02:13.326169 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.326134 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.326359 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.326237 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.326359 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.326347 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km8tl\" (UniqueName: \"kubernetes.io/projected/94e95860-9cd7-4ced-bd39-a38242ff802b-kube-api-access-km8tl\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.326926 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.326905 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.328807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.328787 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.388262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.388182 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km8tl\" (UniqueName: \"kubernetes.io/projected/94e95860-9cd7-4ced-bd39-a38242ff802b-kube-api-access-km8tl\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.483743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.483709 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:13.873046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:13.873013 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7"] Nov 01 01:02:13.877701 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:02:13.877662 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e95860_9cd7_4ced_bd39_a38242ff802b.slice/crio-f779d15211520d4eb72e6a6503634dd8763c66713f6c1e4ffe14c27ae1899a8b WatchSource:0}: Error finding container f779d15211520d4eb72e6a6503634dd8763c66713f6c1e4ffe14c27ae1899a8b: Status 404 returned error can't find the container with id f779d15211520d4eb72e6a6503634dd8763c66713f6c1e4ffe14c27ae1899a8b Nov 01 01:02:14.740883 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:14.740847 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" event={"ID":"94e95860-9cd7-4ced-bd39-a38242ff802b","Type":"ContainerStarted","Data":"805ed5a9bf5f98bf3282db60343e9d459c2795b85cab5acf197d02706d2b4ef8"} Nov 01 01:02:14.741296 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:14.740908 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" event={"ID":"94e95860-9cd7-4ced-bd39-a38242ff802b","Type":"ContainerStarted","Data":"f779d15211520d4eb72e6a6503634dd8763c66713f6c1e4ffe14c27ae1899a8b"} Nov 01 01:02:14.864662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:14.861172 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" podStartSLOduration=1.8611508780000001 podStartE2EDuration="1.861150878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:02:14.857376932 +0000 UTC m=+2324.836959268" watchObservedRunningTime="2026-11-01 01:02:14.861150878 +0000 UTC m=+2324.840733213" Nov 01 01:02:15.751084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:15.751045 4910 generic.go:356] "Generic (PLEG): container finished" podID="94e95860-9cd7-4ced-bd39-a38242ff802b" containerID="805ed5a9bf5f98bf3282db60343e9d459c2795b85cab5acf197d02706d2b4ef8" exitCode=0 Nov 01 01:02:15.751600 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:15.751121 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" event={"ID":"94e95860-9cd7-4ced-bd39-a38242ff802b","Type":"ContainerDied","Data":"805ed5a9bf5f98bf3282db60343e9d459c2795b85cab5acf197d02706d2b4ef8"} Nov 01 01:02:16.940195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.940170 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:16.993961 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.993934 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-xks-ca\") pod \"94e95860-9cd7-4ced-bd39-a38242ff802b\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " Nov 01 01:02:16.994151 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.993980 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-openshift-ca\") pod \"94e95860-9cd7-4ced-bd39-a38242ff802b\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " Nov 01 01:02:16.994151 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.994003 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e95860-9cd7-4ced-bd39-a38242ff802b-kube-api-access-km8tl\" (UniqueName: \"kubernetes.io/projected/94e95860-9cd7-4ced-bd39-a38242ff802b-kube-api-access-km8tl\") pod \"94e95860-9cd7-4ced-bd39-a38242ff802b\" (UID: \"94e95860-9cd7-4ced-bd39-a38242ff802b\") " Nov 01 01:02:16.994749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.994711 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-openshift-ca" pod "94e95860-9cd7-4ced-bd39-a38242ff802b" (UID: "94e95860-9cd7-4ced-bd39-a38242ff802b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:16.994881 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.994855 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:16.996947 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.996919 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e95860-9cd7-4ced-bd39-a38242ff802b-kube-api-access-km8tl" pod "94e95860-9cd7-4ced-bd39-a38242ff802b" (UID: "94e95860-9cd7-4ced-bd39-a38242ff802b"). InnerVolumeSpecName "kube-api-access-km8tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:16.997050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:16.996996 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-xks-ca" pod "94e95860-9cd7-4ced-bd39-a38242ff802b" (UID: "94e95860-9cd7-4ced-bd39-a38242ff802b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:02:17.096868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:17.096772 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e95860-9cd7-4ced-bd39-a38242ff802b-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.096868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:17.096801 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km8tl\" (UniqueName: \"kubernetes.io/projected/94e95860-9cd7-4ced-bd39-a38242ff802b-kube-api-access-km8tl\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:17.774476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:17.774444 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" Nov 01 01:02:17.774686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:17.774476 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7" event={"ID":"94e95860-9cd7-4ced-bd39-a38242ff802b","Type":"ContainerDied","Data":"f779d15211520d4eb72e6a6503634dd8763c66713f6c1e4ffe14c27ae1899a8b"} Nov 01 01:02:17.774686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:17.774528 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f779d15211520d4eb72e6a6503634dd8763c66713f6c1e4ffe14c27ae1899a8b" Nov 01 01:02:25.417828 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.417792 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5"] Nov 01 01:02:25.418686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.418667 4910 cpu_manager.go:414] "containerMap: removing container" podUID="94e95860-9cd7-4ced-bd39-a38242ff802b" containerName="revoke-keys" Nov 01 01:02:25.418686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.418688 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e95860-9cd7-4ced-bd39-a38242ff802b" containerName="revoke-keys" Nov 01 01:02:25.418935 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.418921 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e95860-9cd7-4ced-bd39-a38242ff802b" containerName="revoke-keys" Nov 01 01:02:25.422547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.422526 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.480476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.480444 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5"] Nov 01 01:02:25.481480 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.481453 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.481737 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.481709 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm295\" (UniqueName: \"kubernetes.io/projected/70068e56-d779-46c7-899e-b82506bcb1ae-kube-api-access-nm295\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.481810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.481754 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.524353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.524322 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:02:25.524548 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.524530 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:02:25.582997 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.582963 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm295\" (UniqueName: \"kubernetes.io/projected/70068e56-d779-46c7-899e-b82506bcb1ae-kube-api-access-nm295\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.583168 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.583004 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.583214 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.583177 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.583823 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.583804 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.585679 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.585615 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.644988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.644963 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm295\" (UniqueName: \"kubernetes.io/projected/70068e56-d779-46c7-899e-b82506bcb1ae-kube-api-access-nm295\") pod \"maas-api-revoke-sub-88d5e287-gvjq5\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:25.737516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:25.737399 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:26.276647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:26.251550 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5"] Nov 01 01:02:27.109070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:27.108977 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" event={"ID":"70068e56-d779-46c7-899e-b82506bcb1ae","Type":"ContainerStarted","Data":"e2f3ee4da2edec5de9b026df34599c597325030a0eb31b33bae20544e9b2b3ba"} Nov 01 01:02:27.109070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:27.109035 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" event={"ID":"70068e56-d779-46c7-899e-b82506bcb1ae","Type":"ContainerStarted","Data":"2f1b16d1d7359934c63e320ecd6f784cf9a5a4fbe22f8af08467e8fdfd910287"} Nov 01 01:02:27.226138 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:27.226078 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" podStartSLOduration=2.2260596599999998 podStartE2EDuration="2.22605966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:02:27.224720592 +0000 UTC m=+2337.204302927" watchObservedRunningTime="2026-11-01 01:02:27.22605966 +0000 UTC m=+2337.205641996" Nov 01 01:02:28.118170 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:28.118133 4910 generic.go:356] "Generic (PLEG): container finished" podID="70068e56-d779-46c7-899e-b82506bcb1ae" containerID="e2f3ee4da2edec5de9b026df34599c597325030a0eb31b33bae20544e9b2b3ba" exitCode=0 Nov 01 01:02:28.118578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:28.118218 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" event={"ID":"70068e56-d779-46c7-899e-b82506bcb1ae","Type":"ContainerDied","Data":"e2f3ee4da2edec5de9b026df34599c597325030a0eb31b33bae20544e9b2b3ba"} Nov 01 01:02:29.450315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.450286 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:29.528472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.528441 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-openshift-ca\") pod \"70068e56-d779-46c7-899e-b82506bcb1ae\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " Nov 01 01:02:29.528695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.528486 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70068e56-d779-46c7-899e-b82506bcb1ae-kube-api-access-nm295\" (UniqueName: \"kubernetes.io/projected/70068e56-d779-46c7-899e-b82506bcb1ae-kube-api-access-nm295\") pod \"70068e56-d779-46c7-899e-b82506bcb1ae\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " Nov 01 01:02:29.528695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.528552 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-xks-ca\") pod \"70068e56-d779-46c7-899e-b82506bcb1ae\" (UID: \"70068e56-d779-46c7-899e-b82506bcb1ae\") " Nov 01 01:02:29.529111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.529071 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-openshift-ca" pod "70068e56-d779-46c7-899e-b82506bcb1ae" (UID: "70068e56-d779-46c7-899e-b82506bcb1ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:29.532094 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.532066 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70068e56-d779-46c7-899e-b82506bcb1ae-kube-api-access-nm295" pod "70068e56-d779-46c7-899e-b82506bcb1ae" (UID: "70068e56-d779-46c7-899e-b82506bcb1ae"). InnerVolumeSpecName "kube-api-access-nm295". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:29.532997 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.532979 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-xks-ca" pod "70068e56-d779-46c7-899e-b82506bcb1ae" (UID: "70068e56-d779-46c7-899e-b82506bcb1ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:02:29.634890 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.634844 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:29.634890 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.634885 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm295\" (UniqueName: \"kubernetes.io/projected/70068e56-d779-46c7-899e-b82506bcb1ae-kube-api-access-nm295\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:29.635146 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:29.634902 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70068e56-d779-46c7-899e-b82506bcb1ae-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:30.142291 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:30.142243 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" event={"ID":"70068e56-d779-46c7-899e-b82506bcb1ae","Type":"ContainerDied","Data":"2f1b16d1d7359934c63e320ecd6f784cf9a5a4fbe22f8af08467e8fdfd910287"} Nov 01 01:02:30.142506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:30.142309 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1b16d1d7359934c63e320ecd6f784cf9a5a4fbe22f8af08467e8fdfd910287" Nov 01 01:02:30.142506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:30.142380 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5" Nov 01 01:02:30.423141 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:02:30.423041 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70068e56_d779_46c7_899e_b82506bcb1ae.slice/crio-2f1b16d1d7359934c63e320ecd6f784cf9a5a4fbe22f8af08467e8fdfd910287\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70068e56_d779_46c7_899e_b82506bcb1ae.slice\": RecentStats: unable to find data in memory cache]" Nov 01 01:02:32.028409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:32.028370 4910 scope.go:122] "RemoveContainer" containerID="751cb2ad1b28d8b0116bb1c5ef9376f05d2f8d11779852f6f580f47bfa53129f" Nov 01 01:02:32.041565 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:32.041543 4910 scope.go:122] "RemoveContainer" containerID="ca92f6a6f76c0e724e7842b972bbd338b55536a007578c4538a125bd87867042" Nov 01 01:02:32.124659 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:32.122808 4910 scope.go:122] "RemoveContainer" containerID="ae040e71482d4cabd1b27a60e64e1e7853551b5cf4c6cb9610aa6ea809fcc297" Nov 01 01:02:36.261836 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:36.261738 4910 generic.go:356] "Generic (PLEG): container finished" podID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerID="1fc1fe192823f61332071797fc0848d3933f2f3324f7c856e5b7cfa506a78730" exitCode=0 Nov 01 01:02:36.262263 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:36.261820 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" event={"ID":"a3c237a6-a5ed-488c-85d5-fa0ea4993c65","Type":"ContainerDied","Data":"1fc1fe192823f61332071797fc0848d3933f2f3324f7c856e5b7cfa506a78730"} Nov 01 01:02:36.262371 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:36.262353 4910 scope.go:122] "RemoveContainer" containerID="1fc1fe192823f61332071797fc0848d3933f2f3324f7c856e5b7cfa506a78730" Nov 01 01:02:36.264578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:36.264535 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:02:36.264751 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:36.264675 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:02:36.264751 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:36.264728 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:02:37.271831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.271798 4910 generic.go:356] "Generic (PLEG): container finished" podID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerID="892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0" exitCode=0 Nov 01 01:02:37.272381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.271874 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" event={"ID":"4749198e-b5c8-404f-88ca-f01bd68817a7","Type":"ContainerDied","Data":"892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0"} Nov 01 01:02:37.272381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.272345 4910 scope.go:122] "RemoveContainer" containerID="892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0" Nov 01 01:02:37.274362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.274322 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:02:37.274552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.274408 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:02:37.274552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.274455 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:02:37.274696 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.274548 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" event={"ID":"a3c237a6-a5ed-488c-85d5-fa0ea4993c65","Type":"ContainerStarted","Data":"9cd4661b5fef8a5c8e5c54814333cf68fc4338cc8064a24948c094b2ec3c24fa"} Nov 01 01:02:37.778028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.776991 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:02:37.785649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.781746 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 01:02:37.785649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:37.781804 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 01:02:38.292707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:38.292665 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" event={"ID":"4749198e-b5c8-404f-88ca-f01bd68817a7","Type":"ContainerStarted","Data":"0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c"} Nov 01 01:02:38.777649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:38.774931 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 01:02:38.777649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:38.774988 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 01:02:38.829878 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:38.829848 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:02:38.830135 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:38.830113 4910 patch_prober.go:29] interesting pod/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 01:02:38.830190 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:38.830162 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 01:02:39.783646 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:39.783363 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 01:02:39.783646 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:39.783439 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 01:02:39.830331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:39.830296 4910 patch_prober.go:29] interesting pod/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 01:02:39.830616 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:39.830585 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 01:02:40.181708 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:40.181389 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:02:40.777650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:40.777126 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 01:02:40.777650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:40.777203 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 01:02:40.830977 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:40.830945 4910 patch_prober.go:29] interesting pod/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 01:02:40.831381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:40.831002 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 01:02:41.781801 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:41.781766 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 01:02:41.782113 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:41.782086 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 01:02:41.830808 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:41.830777 4910 patch_prober.go:29] interesting pod/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 01:02:41.830970 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:41.830830 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 01:02:42.778012 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:42.777971 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 01:02:42.778660 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:42.778603 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 01:02:42.830566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:42.830532 4910 patch_prober.go:29] interesting pod/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 01:02:42.830757 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:42.830585 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 01:02:43.118068 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:02:43.117985 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:02:43.789407 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:43.789361 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:02:43.791220 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:43.791193 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:02:43.829999 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:43.829973 4910 patch_prober.go:29] interesting pod/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Nov 01 01:02:43.830131 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:43.830029 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Nov 01 01:02:44.835542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:44.835502 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:02:44.836039 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:44.835971 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:02:44.836749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:44.836728 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:02:53.920978 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:53.920943 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2"] Nov 01 01:02:53.927656 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:53.922169 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" containerID="cri-o://228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c" gracePeriod=30 Nov 01 01:02:56.895111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:56.895071 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:02:56.895543 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:56.895134 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:02:59.217353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.217292 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 01:02:59.304997 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.304967 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-data\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305114 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-credential-socket\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305188 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istiod-ca-cert\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305233 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-data" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:59.305363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305240 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-socket\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305344 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-podinfo\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305473 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305390 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-credential-socket" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:59.305473 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305395 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-token\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-token\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305563 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305475 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-envoy\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305563 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305487 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-socket" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:59.305563 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305537 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-kube-api-access-lmhzm\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-kube-api-access-lmhzm\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305572 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-certs\") pod \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\" (UID: \"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7\") " Nov 01 01:02:59.305787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305577 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istiod-ca-cert" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:02:59.305965 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.305939 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-certs" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:59.306375 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.306339 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.306375 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.306361 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.306545 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.306376 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.306545 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.306390 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.306545 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.306403 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.308136 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.308107 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-envoy" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:02:59.308294 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.308260 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-podinfo" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:02:59.308345 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.308307 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-token" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:59.308584 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.308562 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-kube-api-access-lmhzm" pod "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" (UID: "8c4e0ed5-9803-4d9e-9ac2-b3be268080c7"). InnerVolumeSpecName "kube-api-access-lmhzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:02:59.408282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.408252 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.408282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.408279 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.408282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.408289 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.408569 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.408297 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmhzm\" (UniqueName: \"kubernetes.io/projected/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7-kube-api-access-lmhzm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:02:59.623830 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.623795 4910 generic.go:356] "Generic (PLEG): container finished" podID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerID="228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c" exitCode=0 Nov 01 01:02:59.624026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.623852 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" Nov 01 01:02:59.624026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.623887 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" event={"ID":"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7","Type":"ContainerDied","Data":"228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c"} Nov 01 01:02:59.624026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.623961 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2" event={"ID":"8c4e0ed5-9803-4d9e-9ac2-b3be268080c7","Type":"ContainerDied","Data":"fbcda2736af8f21c2210daa416a2bb4a3f37abf16fd449bfffdbedd23f85b871"} Nov 01 01:02:59.624026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.623983 4910 scope.go:122] "RemoveContainer" containerID="228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c" Nov 01 01:02:59.636225 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.636209 4910 scope.go:122] "RemoveContainer" containerID="228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c" Nov 01 01:02:59.636487 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:02:59.636468 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c\": container with ID starting with 228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c not found: ID does not exist" containerID="228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c" Nov 01 01:02:59.636553 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.636499 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c"} err="failed to get container status \"228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c\": rpc error: code = NotFound desc = could not find container \"228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c\": container with ID starting with 228e0b86b6470caebb473023e17856d105272eaf05217d52a365f75f906ca46c not found: ID does not exist" Nov 01 01:02:59.792984 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.792950 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2"] Nov 01 01:02:59.797193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:02:59.797171 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-6scm2"] Nov 01 01:03:00.195590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:00.195558 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" path="/var/lib/kubelet/pods/8c4e0ed5-9803-4d9e-9ac2-b3be268080c7/volumes" Nov 01 01:03:06.049539 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:06.049504 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9"] Nov 01 01:03:06.101845 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:06.101801 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1847-77zh9"] Nov 01 01:03:06.195816 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:06.195785 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5089845-e580-4c70-8add-bf575c69ac98" path="/var/lib/kubelet/pods/f5089845-e580-4c70-8add-bf575c69ac98/volumes" Nov 01 01:03:13.054700 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:13.053828 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289"] Nov 01 01:03:13.093353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:13.093315 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-27dee6-xc289"] Nov 01 01:03:13.114684 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:03:13.114610 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:03:14.193436 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:14.193402 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d4bab5-2798-4958-9538-b7fd86f5f30c" path="/var/lib/kubelet/pods/20d4bab5-2798-4958-9538-b7fd86f5f30c/volumes" Nov 01 01:03:16.025548 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.025507 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554"] Nov 01 01:03:16.026888 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.026866 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" Nov 01 01:03:16.027021 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.026890 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" Nov 01 01:03:16.027021 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.026920 4910 cpu_manager.go:414] "containerMap: removing container" podUID="70068e56-d779-46c7-899e-b82506bcb1ae" containerName="revoke-keys" Nov 01 01:03:16.027021 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.026928 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70068e56-d779-46c7-899e-b82506bcb1ae" containerName="revoke-keys" Nov 01 01:03:16.027181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.027095 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="70068e56-d779-46c7-899e-b82506bcb1ae" containerName="revoke-keys" Nov 01 01:03:16.027181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.027126 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4e0ed5-9803-4d9e-9ac2-b3be268080c7" containerName="istio-proxy" Nov 01 01:03:16.032322 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.032302 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.088075 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.088042 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554"] Nov 01 01:03:16.131380 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.131349 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:03:16.131559 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.131504 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:03:16.189445 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.189407 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.189647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.189476 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.189647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.189569 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz55\" (UniqueName: \"kubernetes.io/projected/e35669e0-f769-449f-8d3c-ba1e01463219-kube-api-access-npz55\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.291477 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.291403 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.291477 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.291451 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.291755 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.291577 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz55\" (UniqueName: \"kubernetes.io/projected/e35669e0-f769-449f-8d3c-ba1e01463219-kube-api-access-npz55\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.292391 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.292364 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.294293 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.294275 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.375814 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.375786 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz55\" (UniqueName: \"kubernetes.io/projected/e35669e0-f769-449f-8d3c-ba1e01463219-kube-api-access-npz55\") pod \"maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:16.647281 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:16.647200 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:17.079984 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:03:17.079949 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35669e0_f769_449f_8d3c_ba1e01463219.slice/crio-42bb01f7d2c921ab2fcc07ca6d8ae0d4955a80dfa1e4cdb092b0987656206d94 WatchSource:0}: Error finding container 42bb01f7d2c921ab2fcc07ca6d8ae0d4955a80dfa1e4cdb092b0987656206d94: Status 404 returned error can't find the container with id 42bb01f7d2c921ab2fcc07ca6d8ae0d4955a80dfa1e4cdb092b0987656206d94 Nov 01 01:03:17.080355 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:17.080193 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554"] Nov 01 01:03:17.903494 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:17.903455 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" event={"ID":"e35669e0-f769-449f-8d3c-ba1e01463219","Type":"ContainerStarted","Data":"2a9bd4a527741de5f380307d848f8b057b4f8c4e3585c4d9d00467a5bddd4711"} Nov 01 01:03:17.903867 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:17.903842 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" event={"ID":"e35669e0-f769-449f-8d3c-ba1e01463219","Type":"ContainerStarted","Data":"42bb01f7d2c921ab2fcc07ca6d8ae0d4955a80dfa1e4cdb092b0987656206d94"} Nov 01 01:03:18.027320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:18.027266 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" podStartSLOduration=3.027252486 podStartE2EDuration="3.027252486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:03:18.023194448 +0000 UTC m=+2388.002776777" watchObservedRunningTime="2026-11-01 01:03:18.027252486 +0000 UTC m=+2388.006834824" Nov 01 01:03:18.913764 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:18.913728 4910 generic.go:356] "Generic (PLEG): container finished" podID="e35669e0-f769-449f-8d3c-ba1e01463219" containerID="2a9bd4a527741de5f380307d848f8b057b4f8c4e3585c4d9d00467a5bddd4711" exitCode=0 Nov 01 01:03:18.914184 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:18.913840 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" event={"ID":"e35669e0-f769-449f-8d3c-ba1e01463219","Type":"ContainerDied","Data":"2a9bd4a527741de5f380307d848f8b057b4f8c4e3585c4d9d00467a5bddd4711"} Nov 01 01:03:20.163976 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.163949 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:20.281883 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.281848 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-xks-ca\") pod \"e35669e0-f769-449f-8d3c-ba1e01463219\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " Nov 01 01:03:20.282076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.281908 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35669e0-f769-449f-8d3c-ba1e01463219-kube-api-access-npz55\" (UniqueName: \"kubernetes.io/projected/e35669e0-f769-449f-8d3c-ba1e01463219-kube-api-access-npz55\") pod \"e35669e0-f769-449f-8d3c-ba1e01463219\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " Nov 01 01:03:20.282076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.281960 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-openshift-ca\") pod \"e35669e0-f769-449f-8d3c-ba1e01463219\" (UID: \"e35669e0-f769-449f-8d3c-ba1e01463219\") " Nov 01 01:03:20.283111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.282520 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-openshift-ca" pod "e35669e0-f769-449f-8d3c-ba1e01463219" (UID: "e35669e0-f769-449f-8d3c-ba1e01463219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:20.283111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.282884 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:20.309866 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.309797 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-xks-ca" pod "e35669e0-f769-449f-8d3c-ba1e01463219" (UID: "e35669e0-f769-449f-8d3c-ba1e01463219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:20.313990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.313936 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35669e0-f769-449f-8d3c-ba1e01463219-kube-api-access-npz55" pod "e35669e0-f769-449f-8d3c-ba1e01463219" (UID: "e35669e0-f769-449f-8d3c-ba1e01463219"). InnerVolumeSpecName "kube-api-access-npz55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:20.384318 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.384254 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e35669e0-f769-449f-8d3c-ba1e01463219-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:20.384611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.384290 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz55\" (UniqueName: \"kubernetes.io/projected/e35669e0-f769-449f-8d3c-ba1e01463219-kube-api-access-npz55\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:20.934595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.934554 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" event={"ID":"e35669e0-f769-449f-8d3c-ba1e01463219","Type":"ContainerDied","Data":"42bb01f7d2c921ab2fcc07ca6d8ae0d4955a80dfa1e4cdb092b0987656206d94"} Nov 01 01:03:20.934595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.934592 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554" Nov 01 01:03:20.934892 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:20.934616 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42bb01f7d2c921ab2fcc07ca6d8ae0d4955a80dfa1e4cdb092b0987656206d94" Nov 01 01:03:21.153582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.153547 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw"] Nov 01 01:03:21.154369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.154349 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e35669e0-f769-449f-8d3c-ba1e01463219" containerName="revoke-keys" Nov 01 01:03:21.154453 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.154371 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35669e0-f769-449f-8d3c-ba1e01463219" containerName="revoke-keys" Nov 01 01:03:21.154511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.154475 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35669e0-f769-449f-8d3c-ba1e01463219" containerName="revoke-keys" Nov 01 01:03:21.158353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.158327 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.213472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.211931 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:03:21.213472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.212101 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:03:21.213472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.213364 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tfkm\" (UniqueName: \"kubernetes.io/projected/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-kube-api-access-7tfkm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.214045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.213667 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.214045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.213724 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.214167 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.214075 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw"] Nov 01 01:03:21.314819 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.314781 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.314997 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.314834 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.314997 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.314947 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tfkm\" (UniqueName: \"kubernetes.io/projected/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-kube-api-access-7tfkm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.315651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.315558 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.317390 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.317361 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.377120 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.377089 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tfkm\" (UniqueName: \"kubernetes.io/projected/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-kube-api-access-7tfkm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.477561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.477467 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:21.879905 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.879736 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw"] Nov 01 01:03:21.879905 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:03:21.879837 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3ba2f1_bfb0_4cc3_a8e5_c2478a8013c2.slice/crio-03d36fcef513aacdd8642054d7a02a1691e8d50447f89668449d8c4c8f8bd3e8 WatchSource:0}: Error finding container 03d36fcef513aacdd8642054d7a02a1691e8d50447f89668449d8c4c8f8bd3e8: Status 404 returned error can't find the container with id 03d36fcef513aacdd8642054d7a02a1691e8d50447f89668449d8c4c8f8bd3e8 Nov 01 01:03:21.945009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:21.944969 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" event={"ID":"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2","Type":"ContainerStarted","Data":"03d36fcef513aacdd8642054d7a02a1691e8d50447f89668449d8c4c8f8bd3e8"} Nov 01 01:03:23.051774 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:23.050208 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" event={"ID":"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2","Type":"ContainerStarted","Data":"1c083ea601f09fedc3581045407750ecf73c99b07efe205572310e8611b87153"} Nov 01 01:03:23.170700 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:23.170172 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" podStartSLOduration=2.170155546 podStartE2EDuration="2.170155546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:03:23.167930619 +0000 UTC m=+2393.147512955" watchObservedRunningTime="2026-11-01 01:03:23.170155546 +0000 UTC m=+2393.149737881" Nov 01 01:03:24.060372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:24.060318 4910 generic.go:356] "Generic (PLEG): container finished" podID="0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" containerID="1c083ea601f09fedc3581045407750ecf73c99b07efe205572310e8611b87153" exitCode=0 Nov 01 01:03:24.060982 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:24.060738 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" event={"ID":"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2","Type":"ContainerDied","Data":"1c083ea601f09fedc3581045407750ecf73c99b07efe205572310e8611b87153"} Nov 01 01:03:25.157579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.157439 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7"] Nov 01 01:03:25.167680 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.167618 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af82f1-vtxc7"] Nov 01 01:03:25.319544 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.319521 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:25.384250 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.384159 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-xks-ca\") pod \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " Nov 01 01:03:25.384250 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.384239 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-kube-api-access-7tfkm\" (UniqueName: \"kubernetes.io/projected/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-kube-api-access-7tfkm\") pod \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " Nov 01 01:03:25.384497 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.384395 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-openshift-ca\") pod \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\" (UID: \"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2\") " Nov 01 01:03:25.384806 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.384773 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-openshift-ca" pod "0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" (UID: "0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:25.385179 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.385159 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.386553 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.386531 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-xks-ca" pod "0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" (UID: "0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:25.386553 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.386539 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-kube-api-access-7tfkm" pod "0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" (UID: "0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2"). InnerVolumeSpecName "kube-api-access-7tfkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:25.487458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.486932 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:25.487458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:25.486976 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tfkm\" (UniqueName: \"kubernetes.io/projected/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2-kube-api-access-7tfkm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:26.124472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.124442 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" Nov 01 01:03:26.124472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.124457 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw" event={"ID":"0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2","Type":"ContainerDied","Data":"03d36fcef513aacdd8642054d7a02a1691e8d50447f89668449d8c4c8f8bd3e8"} Nov 01 01:03:26.124779 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.124577 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03d36fcef513aacdd8642054d7a02a1691e8d50447f89668449d8c4c8f8bd3e8" Nov 01 01:03:26.126412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.126383 4910 generic.go:356] "Generic (PLEG): container finished" podID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerID="8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8" exitCode=0 Nov 01 01:03:26.126563 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.126454 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" event={"ID":"c8b7ff3e-84d4-4241-81b0-5648a120a61a","Type":"ContainerDied","Data":"8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8"} Nov 01 01:03:26.126884 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.126867 4910 scope.go:122] "RemoveContainer" containerID="8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8" Nov 01 01:03:26.128920 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.128886 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:03:26.129013 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.128979 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:03:26.129071 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.129029 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:03:26.193199 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.193176 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e95860-9cd7-4ced-bd39-a38242ff802b" path="/var/lib/kubelet/pods/94e95860-9cd7-4ced-bd39-a38242ff802b/volumes" Nov 01 01:03:26.342747 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.342713 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9"] Nov 01 01:03:26.343376 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.343359 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" containerName="revoke-keys" Nov 01 01:03:26.343482 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.343380 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" containerName="revoke-keys" Nov 01 01:03:26.343672 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.343572 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" containerName="revoke-keys" Nov 01 01:03:26.347674 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.347460 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.392640 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.392553 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9"] Nov 01 01:03:26.414050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.414020 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:03:26.416541 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.416520 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:03:26.439063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.439030 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4knj\" (UniqueName: \"kubernetes.io/projected/8fb20fe2-ce47-4451-9f66-e767f8b8df18-kube-api-access-g4knj\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.439221 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.439117 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.439425 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.439399 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.540378 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.540346 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.540604 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.540464 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4knj\" (UniqueName: \"kubernetes.io/projected/8fb20fe2-ce47-4451-9f66-e767f8b8df18-kube-api-access-g4knj\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.540604 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.540502 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.541036 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.541018 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.543264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.543235 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.602374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.602341 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4knj\" (UniqueName: \"kubernetes.io/projected/8fb20fe2-ce47-4451-9f66-e767f8b8df18-kube-api-access-g4knj\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:26.668649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:26.668529 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:27.105589 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:03:27.105545 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb20fe2_ce47_4451_9f66_e767f8b8df18.slice/crio-2926343bb55ec4e89321039a1f1dc4d980335b869a55a734ccb6d7567b7810e2 WatchSource:0}: Error finding container 2926343bb55ec4e89321039a1f1dc4d980335b869a55a734ccb6d7567b7810e2: Status 404 returned error can't find the container with id 2926343bb55ec4e89321039a1f1dc4d980335b869a55a734ccb6d7567b7810e2 Nov 01 01:03:27.108034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:27.108007 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9"] Nov 01 01:03:27.139670 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:27.139582 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" event={"ID":"8fb20fe2-ce47-4451-9f66-e767f8b8df18","Type":"ContainerStarted","Data":"2926343bb55ec4e89321039a1f1dc4d980335b869a55a734ccb6d7567b7810e2"} Nov 01 01:03:27.143085 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:27.143063 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" event={"ID":"c8b7ff3e-84d4-4241-81b0-5648a120a61a","Type":"ContainerStarted","Data":"212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe"} Nov 01 01:03:27.865649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:27.860040 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:03:27.865649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:27.860175 4910 patch_prober.go:29] interesting pod/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 01:03:27.865649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:27.860219 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 01:03:28.174898 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:28.174135 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" event={"ID":"8fb20fe2-ce47-4451-9f66-e767f8b8df18","Type":"ContainerStarted","Data":"f4ee5d88f9b7e1731ebbbb6bbfdd767cc45f0187ef0d153e31a5e37ee6abf764"} Nov 01 01:03:28.291721 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:28.291614 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" podStartSLOduration=2.29160105 podStartE2EDuration="2.29160105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:03:28.28916346 +0000 UTC m=+2398.268745795" watchObservedRunningTime="2026-11-01 01:03:28.29160105 +0000 UTC m=+2398.271183385" Nov 01 01:03:28.868650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:28.861689 4910 patch_prober.go:29] interesting pod/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 01:03:28.868650 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:28.861757 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 01:03:29.183547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:29.183459 4910 generic.go:356] "Generic (PLEG): container finished" podID="8fb20fe2-ce47-4451-9f66-e767f8b8df18" containerID="f4ee5d88f9b7e1731ebbbb6bbfdd767cc45f0187ef0d153e31a5e37ee6abf764" exitCode=0 Nov 01 01:03:29.183768 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:29.183546 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" event={"ID":"8fb20fe2-ce47-4451-9f66-e767f8b8df18","Type":"ContainerDied","Data":"f4ee5d88f9b7e1731ebbbb6bbfdd767cc45f0187ef0d153e31a5e37ee6abf764"} Nov 01 01:03:29.857046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:29.857006 4910 patch_prober.go:29] interesting pod/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 01:03:29.857350 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:29.857317 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 01:03:30.445330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.445308 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:30.536806 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.536777 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-xks-ca\") pod \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " Nov 01 01:03:30.536991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.536976 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-openshift-ca\") pod \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " Nov 01 01:03:30.537037 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.537027 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb20fe2-ce47-4451-9f66-e767f8b8df18-kube-api-access-g4knj\" (UniqueName: \"kubernetes.io/projected/8fb20fe2-ce47-4451-9f66-e767f8b8df18-kube-api-access-g4knj\") pod \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\" (UID: \"8fb20fe2-ce47-4451-9f66-e767f8b8df18\") " Nov 01 01:03:30.537405 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.537364 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-openshift-ca" pod "8fb20fe2-ce47-4451-9f66-e767f8b8df18" (UID: "8fb20fe2-ce47-4451-9f66-e767f8b8df18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:30.537734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.537520 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:30.539201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.539170 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-xks-ca" pod "8fb20fe2-ce47-4451-9f66-e767f8b8df18" (UID: "8fb20fe2-ce47-4451-9f66-e767f8b8df18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:30.539380 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.539358 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb20fe2-ce47-4451-9f66-e767f8b8df18-kube-api-access-g4knj" pod "8fb20fe2-ce47-4451-9f66-e767f8b8df18" (UID: "8fb20fe2-ce47-4451-9f66-e767f8b8df18"). InnerVolumeSpecName "kube-api-access-g4knj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:30.638672 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.638614 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb20fe2-ce47-4451-9f66-e767f8b8df18-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:30.638672 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.638659 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4knj\" (UniqueName: \"kubernetes.io/projected/8fb20fe2-ce47-4451-9f66-e767f8b8df18-kube-api-access-g4knj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:30.733776 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.733750 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:03:30.780658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.775309 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:03:30.794141 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.794071 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:03:30.808694 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.808671 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:03:30.876648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.876163 4910 patch_prober.go:29] interesting pod/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 01:03:30.876648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:30.876241 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 01:03:31.243796 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:31.243720 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" Nov 01 01:03:31.243796 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:31.243756 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9" event={"ID":"8fb20fe2-ce47-4451-9f66-e767f8b8df18","Type":"ContainerDied","Data":"2926343bb55ec4e89321039a1f1dc4d980335b869a55a734ccb6d7567b7810e2"} Nov 01 01:03:31.243977 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:31.243803 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2926343bb55ec4e89321039a1f1dc4d980335b869a55a734ccb6d7567b7810e2" Nov 01 01:03:31.858649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:31.856937 4910 patch_prober.go:29] interesting pod/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 01 01:03:31.858649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:31.856993 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 01 01:03:32.280943 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:32.280910 4910 scope.go:122] "RemoveContainer" containerID="4e6a4635c283b3454d94799ace9fa1df3e41d7eaef47f12235cb431c2513a2f1" Nov 01 01:03:32.988659 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:32.987993 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:03:33.082657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.078368 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:03:33.090658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.086743 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:03:33.548659 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.548608 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c"] Nov 01 01:03:33.549360 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.549344 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8fb20fe2-ce47-4451-9f66-e767f8b8df18" containerName="revoke-keys" Nov 01 01:03:33.549428 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.549361 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb20fe2-ce47-4451-9f66-e767f8b8df18" containerName="revoke-keys" Nov 01 01:03:33.549490 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.549480 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb20fe2-ce47-4451-9f66-e767f8b8df18" containerName="revoke-keys" Nov 01 01:03:33.554900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.554873 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.609315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.609265 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:03:33.610474 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.610454 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:03:33.614500 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.614476 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c"] Nov 01 01:03:33.694023 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.693991 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.694212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.694192 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqfxk\" (UniqueName: \"kubernetes.io/projected/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-kube-api-access-pqfxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.694328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.694309 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.795971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.795934 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqfxk\" (UniqueName: \"kubernetes.io/projected/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-kube-api-access-pqfxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.796179 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.796064 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.796246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.796191 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.796975 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.796946 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.798941 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.798880 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.857426 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.857393 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqfxk\" (UniqueName: \"kubernetes.io/projected/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-kube-api-access-pqfxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:33.874448 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:33.874421 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:34.712390 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:34.712359 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c"] Nov 01 01:03:34.714363 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:03:34.714335 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b27b78_ad84_4929_a8f2_8a6d7e8c6ca3.slice/crio-cdda620d23e21afc5d1d1060b2b9e833f4bb89a67d53a9756bf64471d31e50e5 WatchSource:0}: Error finding container cdda620d23e21afc5d1d1060b2b9e833f4bb89a67d53a9756bf64471d31e50e5: Status 404 returned error can't find the container with id cdda620d23e21afc5d1d1060b2b9e833f4bb89a67d53a9756bf64471d31e50e5 Nov 01 01:03:35.360067 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:35.360027 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" event={"ID":"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3","Type":"ContainerStarted","Data":"41b0d1c8c0903dae110f820bf5bac2c445e6b81899e765fe59c08280f97a6184"} Nov 01 01:03:35.360253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:35.360094 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" event={"ID":"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3","Type":"ContainerStarted","Data":"cdda620d23e21afc5d1d1060b2b9e833f4bb89a67d53a9756bf64471d31e50e5"} Nov 01 01:03:35.486617 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:35.485179 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" podStartSLOduration=2.485156866 podStartE2EDuration="2.485156866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:03:35.477191539 +0000 UTC m=+2405.456773875" watchObservedRunningTime="2026-11-01 01:03:35.485156866 +0000 UTC m=+2405.464739202" Nov 01 01:03:36.372029 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:36.371992 4910 generic.go:356] "Generic (PLEG): container finished" podID="79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" containerID="41b0d1c8c0903dae110f820bf5bac2c445e6b81899e765fe59c08280f97a6184" exitCode=0 Nov 01 01:03:36.372465 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:36.372059 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" event={"ID":"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3","Type":"ContainerDied","Data":"41b0d1c8c0903dae110f820bf5bac2c445e6b81899e765fe59c08280f97a6184"} Nov 01 01:03:37.548796 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.548767 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:37.594960 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.594933 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-kube-api-access-pqfxk\" (UniqueName: \"kubernetes.io/projected/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-kube-api-access-pqfxk\") pod \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " Nov 01 01:03:37.595116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.595049 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-xks-ca\") pod \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " Nov 01 01:03:37.595187 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.595115 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-openshift-ca\") pod \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\" (UID: \"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3\") " Nov 01 01:03:37.595530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.595500 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-openshift-ca" pod "79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" (UID: "79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:03:37.596040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.596017 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:37.597315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.597279 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-xks-ca" pod "79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" (UID: "79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:03:37.597421 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.597286 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-kube-api-access-pqfxk" pod "79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" (UID: "79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3"). InnerVolumeSpecName "kube-api-access-pqfxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:03:37.697348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.697259 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqfxk\" (UniqueName: \"kubernetes.io/projected/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-kube-api-access-pqfxk\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:37.697348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:37.697289 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:03:38.403978 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:38.403940 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" event={"ID":"79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3","Type":"ContainerDied","Data":"cdda620d23e21afc5d1d1060b2b9e833f4bb89a67d53a9756bf64471d31e50e5"} Nov 01 01:03:38.404156 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:38.404005 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdda620d23e21afc5d1d1060b2b9e833f4bb89a67d53a9756bf64471d31e50e5" Nov 01 01:03:38.404156 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:38.403957 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c" Nov 01 01:03:43.107105 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:03:43.107067 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:03:52.880394 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:52.880352 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c"] Nov 01 01:03:52.929461 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:52.929425 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc241b-wpw4c"] Nov 01 01:03:54.197520 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:54.197486 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" path="/var/lib/kubelet/pods/79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3/volumes" Nov 01 01:03:58.813463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:58.813431 4910 generic.go:356] "Generic (PLEG): container finished" podID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerID="e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579" exitCode=0 Nov 01 01:03:58.813982 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:58.813500 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" event={"ID":"d54137c0-9741-4bf6-8c56-a57244f6d7db","Type":"ContainerDied","Data":"e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579"} Nov 01 01:03:58.813982 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:58.813910 4910 scope.go:122] "RemoveContainer" containerID="e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579" Nov 01 01:03:58.815483 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:58.815449 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:03:58.815596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:58.815539 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:03:58.815596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:58.815588 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:03:59.360232 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:59.360199 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx"] Nov 01 01:03:59.365303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:59.365277 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-xzcsx"] Nov 01 01:03:59.823593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:03:59.823556 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" event={"ID":"d54137c0-9741-4bf6-8c56-a57244f6d7db","Type":"ContainerStarted","Data":"976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e"} Nov 01 01:04:00.200949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:00.200862 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de11323-a34f-4b75-80bf-4273146dbb90" path="/var/lib/kubelet/pods/2de11323-a34f-4b75-80bf-4273146dbb90/volumes" Nov 01 01:04:00.394654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:00.375400 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:04:00.394654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:00.375615 4910 patch_prober.go:29] interesting pod/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 01:04:00.394654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:00.375705 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 01:04:00.400378 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:00.400340 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9"] Nov 01 01:04:00.400929 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:00.400876 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" containerID="cri-o://790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406" gracePeriod=30 Nov 01 01:04:01.377165 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:01.376723 4910 patch_prober.go:29] interesting pod/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 01:04:01.377165 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:01.376784 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 01:04:02.358370 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:02.358334 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:04:02.358703 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:02.358674 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:04:02.373976 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:02.373949 4910 patch_prober.go:29] interesting pod/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 01:04:02.374234 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:02.374208 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 01:04:03.374565 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:03.374525 4910 patch_prober.go:29] interesting pod/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 01:04:03.375189 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:03.375158 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 01:04:04.378025 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.377981 4910 patch_prober.go:29] interesting pod/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 01:04:04.378663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.378610 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 01:04:04.747374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.747334 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc"] Nov 01 01:04:04.748910 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.748884 4910 cpu_manager.go:414] "containerMap: removing container" podUID="79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" containerName="revoke-keys" Nov 01 01:04:04.748910 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.748911 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" containerName="revoke-keys" Nov 01 01:04:04.749265 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.749248 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b27b78-ad84-4929-a8f2-8a6d7e8c6ca3" containerName="revoke-keys" Nov 01 01:04:04.787715 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.787677 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:04.806094 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.805508 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc"] Nov 01 01:04:04.840280 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.840254 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:04:04.858834 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.858806 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:04:04.928688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.928649 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:04.929275 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.929247 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzztg\" (UniqueName: \"kubernetes.io/projected/ee312436-e2ce-4946-bf21-a8914942ed47-kube-api-access-jzztg\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:04.929507 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:04.929490 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.031929 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.031288 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzztg\" (UniqueName: \"kubernetes.io/projected/ee312436-e2ce-4946-bf21-a8914942ed47-kube-api-access-jzztg\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.031929 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.031389 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.031929 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.031423 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.032348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.032330 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.034218 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.034194 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.093327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.093287 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzztg\" (UniqueName: \"kubernetes.io/projected/ee312436-e2ce-4946-bf21-a8914942ed47-kube-api-access-jzztg\") pod \"maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.109849 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.109820 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:05.373709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.372781 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:04:05.383678 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.381769 4910 patch_prober.go:29] interesting pod/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 01 01:04:05.383678 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.381839 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 01 01:04:05.527944 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:04:05.527907 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee312436_e2ce_4946_bf21_a8914942ed47.slice/crio-997ea2d1b72aafcfd905ab5131540f6ad941d237143c50696d5784f8a120cb53 WatchSource:0}: Error finding container 997ea2d1b72aafcfd905ab5131540f6ad941d237143c50696d5784f8a120cb53: Status 404 returned error can't find the container with id 997ea2d1b72aafcfd905ab5131540f6ad941d237143c50696d5784f8a120cb53 Nov 01 01:04:05.528463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.528441 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc"] Nov 01 01:04:05.709458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.709433 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 01:04:05.845472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.845437 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-envoy\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.845675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.845656 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-kube-api-access-6xdx6\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-kube-api-access-6xdx6\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.845735 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.845724 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-credential-socket\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.845816 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.845795 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44489bec-b295-4631-ae02-50ed5f97a2cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44489bec-b295-4631-ae02-50ed5f97a2cd-istiod-ca-cert\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.845876 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.845838 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-data\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.845949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.845933 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-certs\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.846008 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.845970 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-token\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-token\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.846063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.846030 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-podinfo\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.846123 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.846080 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-socket\") pod \"44489bec-b295-4631-ae02-50ed5f97a2cd\" (UID: \"44489bec-b295-4631-ae02-50ed5f97a2cd\") " Nov 01 01:04:05.846185 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.846117 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-credential-socket" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:05.846749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.846722 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.846893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.846746 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-certs" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:05.846972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.846937 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-data" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:05.846972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.846961 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-socket" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:05.847101 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.847012 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44489bec-b295-4631-ae02-50ed5f97a2cd-istiod-ca-cert" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:04:05.849351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.849325 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-envoy" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:05.849457 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.849361 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-token" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:05.849596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.849579 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-kube-api-access-6xdx6" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "kube-api-access-6xdx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:05.851098 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.851061 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-podinfo" pod "44489bec-b295-4631-ae02-50ed5f97a2cd" (UID: "44489bec-b295-4631-ae02-50ed5f97a2cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:04:05.947736 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947664 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44489bec-b295-4631-ae02-50ed5f97a2cd-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.947736 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947694 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.947736 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947705 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.947736 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947716 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.947736 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947728 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.948012 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947739 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.948012 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947752 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44489bec-b295-4631-ae02-50ed5f97a2cd-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.948012 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.947763 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xdx6\" (UniqueName: \"kubernetes.io/projected/44489bec-b295-4631-ae02-50ed5f97a2cd-kube-api-access-6xdx6\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:05.949477 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.949446 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" event={"ID":"ee312436-e2ce-4946-bf21-a8914942ed47","Type":"ContainerStarted","Data":"997ea2d1b72aafcfd905ab5131540f6ad941d237143c50696d5784f8a120cb53"} Nov 01 01:04:05.952380 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.952352 4910 generic.go:356] "Generic (PLEG): container finished" podID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerID="790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406" exitCode=0 Nov 01 01:04:05.953219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.952418 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" Nov 01 01:04:05.953219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.952427 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" event={"ID":"44489bec-b295-4631-ae02-50ed5f97a2cd","Type":"ContainerDied","Data":"790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406"} Nov 01 01:04:05.953219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.952486 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9" event={"ID":"44489bec-b295-4631-ae02-50ed5f97a2cd","Type":"ContainerDied","Data":"d945642915e416beca0f13aa86c781616e5fcc73610691fd10e7c2dc64d2179b"} Nov 01 01:04:05.953219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.952516 4910 scope.go:122] "RemoveContainer" containerID="790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406" Nov 01 01:04:05.968776 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.968725 4910 scope.go:122] "RemoveContainer" containerID="4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a" Nov 01 01:04:05.979511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.979484 4910 scope.go:122] "RemoveContainer" containerID="790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406" Nov 01 01:04:05.979841 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:04:05.979820 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406\": container with ID starting with 790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406 not found: ID does not exist" containerID="790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406" Nov 01 01:04:05.979911 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.979856 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406"} err="failed to get container status \"790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406\": rpc error: code = NotFound desc = could not find container \"790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406\": container with ID starting with 790056a2d59eb88844af8dd99587924e75b068feb9aac7afa2d5252cc320f406 not found: ID does not exist" Nov 01 01:04:05.979911 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.979881 4910 scope.go:122] "RemoveContainer" containerID="4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a" Nov 01 01:04:05.980134 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:04:05.980115 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a\": container with ID starting with 4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a not found: ID does not exist" containerID="4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a" Nov 01 01:04:05.980185 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:05.980140 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a"} err="failed to get container status \"4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a\": rpc error: code = NotFound desc = could not find container \"4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a\": container with ID starting with 4d94024efc133f241558e26b181521042a8051000e1b232739cf00cdcaa3eb9a not found: ID does not exist" Nov 01 01:04:06.121146 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:06.121116 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9"] Nov 01 01:04:06.127123 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:06.127097 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bc241b-openshift-default-56bd5bfd4f-gwrx9"] Nov 01 01:04:06.196482 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:06.196448 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" path="/var/lib/kubelet/pods/44489bec-b295-4631-ae02-50ed5f97a2cd/volumes" Nov 01 01:04:06.379097 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:06.379064 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:04:06.380085 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:06.380065 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:04:06.967306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:06.967213 4910 generic.go:356] "Generic (PLEG): container finished" podID="ee312436-e2ce-4946-bf21-a8914942ed47" containerID="64f9336e6f1bcc5047cd16c943d9396a7d4b598df4280bf07796bd0732687b64" exitCode=0 Nov 01 01:04:06.967306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:06.967281 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" event={"ID":"ee312436-e2ce-4946-bf21-a8914942ed47","Type":"ContainerDied","Data":"64f9336e6f1bcc5047cd16c943d9396a7d4b598df4280bf07796bd0732687b64"} Nov 01 01:04:08.139895 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.139870 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:08.283444 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.283411 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-xks-ca\") pod \"ee312436-e2ce-4946-bf21-a8914942ed47\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " Nov 01 01:04:08.283704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.283678 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee312436-e2ce-4946-bf21-a8914942ed47-kube-api-access-jzztg\" (UniqueName: \"kubernetes.io/projected/ee312436-e2ce-4946-bf21-a8914942ed47-kube-api-access-jzztg\") pod \"ee312436-e2ce-4946-bf21-a8914942ed47\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " Nov 01 01:04:08.283830 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.283738 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-openshift-ca\") pod \"ee312436-e2ce-4946-bf21-a8914942ed47\" (UID: \"ee312436-e2ce-4946-bf21-a8914942ed47\") " Nov 01 01:04:08.284706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.284679 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-openshift-ca" pod "ee312436-e2ce-4946-bf21-a8914942ed47" (UID: "ee312436-e2ce-4946-bf21-a8914942ed47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:04:08.286235 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.286204 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-xks-ca" pod "ee312436-e2ce-4946-bf21-a8914942ed47" (UID: "ee312436-e2ce-4946-bf21-a8914942ed47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:04:08.286334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.286284 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee312436-e2ce-4946-bf21-a8914942ed47-kube-api-access-jzztg" pod "ee312436-e2ce-4946-bf21-a8914942ed47" (UID: "ee312436-e2ce-4946-bf21-a8914942ed47"). InnerVolumeSpecName "kube-api-access-jzztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:08.385726 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.385694 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzztg\" (UniqueName: \"kubernetes.io/projected/ee312436-e2ce-4946-bf21-a8914942ed47-kube-api-access-jzztg\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:08.385726 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.385717 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:08.385726 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.385726 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee312436-e2ce-4946-bf21-a8914942ed47-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:08.991170 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.991138 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" Nov 01 01:04:08.991351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.991166 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc" event={"ID":"ee312436-e2ce-4946-bf21-a8914942ed47","Type":"ContainerDied","Data":"997ea2d1b72aafcfd905ab5131540f6ad941d237143c50696d5784f8a120cb53"} Nov 01 01:04:08.991351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:08.991216 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997ea2d1b72aafcfd905ab5131540f6ad941d237143c50696d5784f8a120cb53" Nov 01 01:04:13.104392 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:04:13.104354 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:04:24.622657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:24.622601 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx"] Nov 01 01:04:24.623070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:24.622828 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" containerID="cri-o://91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7" gracePeriod=30 Nov 01 01:04:28.111016 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:28.110980 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:04:28.111435 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:28.111049 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:04:29.914540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:29.914514 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 01:04:30.022725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.022695 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-envoy\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.022893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.022752 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-podinfo\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.022893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.022887 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istiod-ca-cert\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.022988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.022917 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-token\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-token\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.023339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.023303 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istiod-ca-cert" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:04:30.023706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.023426 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-socket\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.023706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.023489 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-kube-api-access-pgwhb\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-kube-api-access-pgwhb\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.023706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.023550 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-credential-socket\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.023706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.023605 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-certs\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.023981 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.023766 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-data\") pod \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\" (UID: \"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab\") " Nov 01 01:04:30.024081 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024055 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-socket" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:30.024337 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024311 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-certs" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:30.024449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024345 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-data" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:30.024611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024592 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-credential-socket" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:30.024725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024710 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.024779 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024733 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.024779 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024747 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.024779 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.024762 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.025711 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.025686 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-podinfo" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:04:30.025827 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.025759 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-envoy" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:30.026015 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.025997 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-token" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:30.027011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.026991 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-kube-api-access-pgwhb" pod "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" (UID: "b0f37726-d65c-4fdc-9f81-1ca081a7b8ab"). InnerVolumeSpecName "kube-api-access-pgwhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:30.126273 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.126236 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.126273 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.126273 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.126534 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.126287 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.126534 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.126301 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwhb\" (UniqueName: \"kubernetes.io/projected/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-kube-api-access-pgwhb\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.126534 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.126315 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:30.337577 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.337538 4910 generic.go:356] "Generic (PLEG): container finished" podID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerID="91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7" exitCode=0 Nov 01 01:04:30.337813 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.337590 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" Nov 01 01:04:30.337813 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.337614 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" event={"ID":"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab","Type":"ContainerDied","Data":"91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7"} Nov 01 01:04:30.337813 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.337707 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx" event={"ID":"b0f37726-d65c-4fdc-9f81-1ca081a7b8ab","Type":"ContainerDied","Data":"5993a73a70708067824ff4de47809682002e34e47255d41fa41c92e036ac5396"} Nov 01 01:04:30.337813 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.337729 4910 scope.go:122] "RemoveContainer" containerID="91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7" Nov 01 01:04:30.352938 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.352918 4910 scope.go:122] "RemoveContainer" containerID="91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7" Nov 01 01:04:30.353419 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:04:30.353330 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7\": container with ID starting with 91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7 not found: ID does not exist" containerID="91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7" Nov 01 01:04:30.353419 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.353367 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7"} err="failed to get container status \"91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7\": rpc error: code = NotFound desc = could not find container \"91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7\": container with ID starting with 91f245b672cbe02c9b7eaa72c4e29133e0ffb58ac1f4b35811ff4e421eec18c7 not found: ID does not exist" Nov 01 01:04:30.505690 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.505654 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx"] Nov 01 01:04:30.555747 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:30.555700 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-7zsnx"] Nov 01 01:04:32.194453 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:32.194421 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" path="/var/lib/kubelet/pods/b0f37726-d65c-4fdc-9f81-1ca081a7b8ab/volumes" Nov 01 01:04:32.322026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:32.321999 4910 scope.go:122] "RemoveContainer" containerID="b3aaf8d9f6a728ec5eacda825e2d8d2d1c2e466fdc7e89538a1381599519ee20" Nov 01 01:04:43.109045 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:04:43.108960 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:04:43.456267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:43.456186 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj"] Nov 01 01:04:43.456432 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:43.456401 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" containerID="cri-o://e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e" gracePeriod=30 Nov 01 01:04:45.231421 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:45.231386 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:04:45.231822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:45.231450 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:04:47.836133 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:47.834896 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9"] Nov 01 01:04:47.907166 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:47.907127 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ba5b10-fqnv9"] Nov 01 01:04:48.194402 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.194311 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb20fe2-ce47-4451-9f66-e767f8b8df18" path="/var/lib/kubelet/pods/8fb20fe2-ce47-4451-9f66-e767f8b8df18/volumes" Nov 01 01:04:48.746977 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.746953 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 01:04:48.821380 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821349 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-certs\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.821542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821387 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-data\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.821542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821439 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-envoy\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.821542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821496 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-socket\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.821782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821558 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istiod-ca-cert\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.821782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821596 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-certs" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:48.821782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821651 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-token\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-token\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.821782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821730 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-data" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:48.821782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821735 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-kube-api-access-mx6qw\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-kube-api-access-mx6qw\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.822086 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821850 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-credential-socket\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.822086 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821878 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-socket" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:48.822086 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.821918 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-podinfo\") pod \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\" (UID: \"20e0c83e-890c-4a87-888e-93c1f5bbc00e\") " Nov 01 01:04:48.822265 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.822103 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istiod-ca-cert" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:04:48.822265 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.822136 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-credential-socket" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:48.822732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.822528 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.822732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.822558 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.822732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.822573 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.822732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.822582 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.822732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.822590 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.824214 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.824186 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-envoy" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:04:48.824325 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.824278 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-podinfo" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:04:48.824545 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.824527 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-token" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:48.824591 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.824557 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-kube-api-access-mx6qw" pod "20e0c83e-890c-4a87-888e-93c1f5bbc00e" (UID: "20e0c83e-890c-4a87-888e-93c1f5bbc00e"). InnerVolumeSpecName "kube-api-access-mx6qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:48.924339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.924306 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.924339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.924334 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.924339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.924344 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6qw\" (UniqueName: \"kubernetes.io/projected/20e0c83e-890c-4a87-888e-93c1f5bbc00e-kube-api-access-mx6qw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:48.924976 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:48.924354 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e0c83e-890c-4a87-888e-93c1f5bbc00e-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:49.230397 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.230366 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554"] Nov 01 01:04:49.234462 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.234435 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f9d81-cs554"] Nov 01 01:04:49.511747 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.511713 4910 generic.go:356] "Generic (PLEG): container finished" podID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerID="e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e" exitCode=0 Nov 01 01:04:49.512142 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.512114 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" event={"ID":"20e0c83e-890c-4a87-888e-93c1f5bbc00e","Type":"ContainerDied","Data":"e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e"} Nov 01 01:04:49.512286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.512168 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" event={"ID":"20e0c83e-890c-4a87-888e-93c1f5bbc00e","Type":"ContainerDied","Data":"af3673ef3c91203201f5a43267feec517dc685c226b2d43dac6c847acef2f299"} Nov 01 01:04:49.512286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.512198 4910 scope.go:122] "RemoveContainer" containerID="e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e" Nov 01 01:04:49.512286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.512061 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj" Nov 01 01:04:49.527732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.527710 4910 scope.go:122] "RemoveContainer" containerID="e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e" Nov 01 01:04:49.528053 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:04:49.528033 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e\": container with ID starting with e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e not found: ID does not exist" containerID="e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e" Nov 01 01:04:49.528145 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.528060 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e"} err="failed to get container status \"e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e\": rpc error: code = NotFound desc = could not find container \"e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e\": container with ID starting with e22291221a15fccc4e4bb7c8aea49de9bf680a3106e9fefe3081fabb494cc15e not found: ID does not exist" Nov 01 01:04:49.683400 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.683371 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj"] Nov 01 01:04:49.689806 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:49.689771 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-msvsj"] Nov 01 01:04:50.196077 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:50.196045 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" path="/var/lib/kubelet/pods/20e0c83e-890c-4a87-888e-93c1f5bbc00e/volumes" Nov 01 01:04:50.196681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:50.196662 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35669e0-f769-449f-8d3c-ba1e01463219" path="/var/lib/kubelet/pods/e35669e0-f769-449f-8d3c-ba1e01463219/volumes" Nov 01 01:04:51.581421 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.581386 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh"] Nov 01 01:04:51.582366 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582347 4910 cpu_manager.go:414] "containerMap: removing container" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" Nov 01 01:04:51.582416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582370 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" Nov 01 01:04:51.582416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582393 4910 cpu_manager.go:414] "containerMap: removing container" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" Nov 01 01:04:51.582416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582401 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" Nov 01 01:04:51.582511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582442 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ee312436-e2ce-4946-bf21-a8914942ed47" containerName="revoke-keys" Nov 01 01:04:51.582511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582449 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee312436-e2ce-4946-bf21-a8914942ed47" containerName="revoke-keys" Nov 01 01:04:51.582511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582468 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" Nov 01 01:04:51.582511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582475 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" Nov 01 01:04:51.582661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582615 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f37726-d65c-4fdc-9f81-1ca081a7b8ab" containerName="istio-proxy" Nov 01 01:04:51.582661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582654 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" Nov 01 01:04:51.582735 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582665 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e0c83e-890c-4a87-888e-93c1f5bbc00e" containerName="istio-proxy" Nov 01 01:04:51.582735 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582678 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" Nov 01 01:04:51.582735 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.582687 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee312436-e2ce-4946-bf21-a8914942ed47" containerName="revoke-keys" Nov 01 01:04:51.588325 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.588306 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.640942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.640910 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh"] Nov 01 01:04:51.661394 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.661354 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/8f777449-8ed0-419e-afd6-a3ffce290448-kube-api-access-sdwgw\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.661583 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.661502 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.661673 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.661610 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.686448 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.686402 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:04:51.686448 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.686429 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:04:51.762578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.762553 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.762790 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.762772 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/8f777449-8ed0-419e-afd6-a3ffce290448-kube-api-access-sdwgw\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.762886 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.762871 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.763675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.763650 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.765347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.765318 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.824116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.824090 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/8f777449-8ed0-419e-afd6-a3ffce290448-kube-api-access-sdwgw\") pod \"maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:51.906900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:51.906825 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:52.323235 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:52.323132 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh"] Nov 01 01:04:52.543708 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:52.543672 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" event={"ID":"8f777449-8ed0-419e-afd6-a3ffce290448","Type":"ContainerStarted","Data":"7cec37e41784a983bc082fa31731ea39ef14a3b12e61869a8caf40d192f69efb"} Nov 01 01:04:53.560391 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:53.560359 4910 generic.go:356] "Generic (PLEG): container finished" podID="8f777449-8ed0-419e-afd6-a3ffce290448" containerID="353328333bf875195a25a751996773934c6aeaadec14984b22a719626ccfc0ad" exitCode=0 Nov 01 01:04:53.560892 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:53.560616 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" event={"ID":"8f777449-8ed0-419e-afd6-a3ffce290448","Type":"ContainerDied","Data":"353328333bf875195a25a751996773934c6aeaadec14984b22a719626ccfc0ad"} Nov 01 01:04:55.111111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.111086 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:55.181534 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.181505 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f777449-8ed0-419e-afd6-a3ffce290448-kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/8f777449-8ed0-419e-afd6-a3ffce290448-kube-api-access-sdwgw\") pod \"8f777449-8ed0-419e-afd6-a3ffce290448\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " Nov 01 01:04:55.181747 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.181613 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-xks-ca\") pod \"8f777449-8ed0-419e-afd6-a3ffce290448\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " Nov 01 01:04:55.181891 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.181869 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-openshift-ca\") pod \"8f777449-8ed0-419e-afd6-a3ffce290448\" (UID: \"8f777449-8ed0-419e-afd6-a3ffce290448\") " Nov 01 01:04:55.182574 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.182545 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-openshift-ca" pod "8f777449-8ed0-419e-afd6-a3ffce290448" (UID: "8f777449-8ed0-419e-afd6-a3ffce290448"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:04:55.183944 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.183923 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f777449-8ed0-419e-afd6-a3ffce290448-kube-api-access-sdwgw" pod "8f777449-8ed0-419e-afd6-a3ffce290448" (UID: "8f777449-8ed0-419e-afd6-a3ffce290448"). InnerVolumeSpecName "kube-api-access-sdwgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:04:55.184050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.183964 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-xks-ca" pod "8f777449-8ed0-419e-afd6-a3ffce290448" (UID: "8f777449-8ed0-419e-afd6-a3ffce290448"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:04:55.283956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.283926 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:55.283956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.283956 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/8f777449-8ed0-419e-afd6-a3ffce290448-kube-api-access-sdwgw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:55.284212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.283972 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f777449-8ed0-419e-afd6-a3ffce290448-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:04:55.691791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.691748 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" event={"ID":"8f777449-8ed0-419e-afd6-a3ffce290448","Type":"ContainerDied","Data":"7cec37e41784a983bc082fa31731ea39ef14a3b12e61869a8caf40d192f69efb"} Nov 01 01:04:55.692024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.691816 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cec37e41784a983bc082fa31731ea39ef14a3b12e61869a8caf40d192f69efb" Nov 01 01:04:55.692024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:55.691890 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh" Nov 01 01:04:56.078881 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.078838 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw"] Nov 01 01:04:56.082946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.082919 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e296f9-sh7xw"] Nov 01 01:04:56.193016 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.192987 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2" path="/var/lib/kubelet/pods/0f3ba2f1-bfb0-4cc3-a8e5-c2478a8013c2/volumes" Nov 01 01:04:56.484552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.484479 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h"] Nov 01 01:04:56.485131 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.485118 4910 cpu_manager.go:414] "containerMap: removing container" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" Nov 01 01:04:56.485227 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.485133 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44489bec-b295-4631-ae02-50ed5f97a2cd" containerName="istio-proxy" Nov 01 01:04:56.485227 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.485144 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8f777449-8ed0-419e-afd6-a3ffce290448" containerName="revoke-keys" Nov 01 01:04:56.485227 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.485150 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f777449-8ed0-419e-afd6-a3ffce290448" containerName="revoke-keys" Nov 01 01:04:56.485332 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.485246 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f777449-8ed0-419e-afd6-a3ffce290448" containerName="revoke-keys" Nov 01 01:04:56.489094 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.489071 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.519206 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.519173 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9jt\" (UniqueName: \"kubernetes.io/projected/7112b666-6fe9-49f0-bc72-9433bedd4abb-kube-api-access-qf9jt\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.519415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.519398 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.519483 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.519432 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.549794 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.549771 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:04:56.550759 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.549843 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:04:56.552958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.552932 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h"] Nov 01 01:04:56.620938 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.620902 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9jt\" (UniqueName: \"kubernetes.io/projected/7112b666-6fe9-49f0-bc72-9433bedd4abb-kube-api-access-qf9jt\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.621186 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.621165 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.621272 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.621204 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.622079 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.622054 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.623777 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.623758 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.682722 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.682689 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9jt\" (UniqueName: \"kubernetes.io/projected/7112b666-6fe9-49f0-bc72-9433bedd4abb-kube-api-access-qf9jt\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:56.782161 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.782127 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd"] Nov 01 01:04:56.782420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.782387 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" containerID="cri-o://0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c" gracePeriod=30 Nov 01 01:04:56.803351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:56.803322 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:04:57.242293 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:04:57.242262 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7112b666_6fe9_49f0_bc72_9433bedd4abb.slice/crio-d08dca64dc86c23849fd26b033bf05c79d1a7065de7877fa26a1dd4d8083e5b4 WatchSource:0}: Error finding container d08dca64dc86c23849fd26b033bf05c79d1a7065de7877fa26a1dd4d8083e5b4: Status 404 returned error can't find the container with id d08dca64dc86c23849fd26b033bf05c79d1a7065de7877fa26a1dd4d8083e5b4 Nov 01 01:04:57.242783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:57.242364 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h"] Nov 01 01:04:57.710947 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:57.710910 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" event={"ID":"7112b666-6fe9-49f0-bc72-9433bedd4abb","Type":"ContainerStarted","Data":"35dd2d3a62cdd7746ce61bb6f2abc80fabe17b9fcf5a92d427add5f2526fb6f7"} Nov 01 01:04:57.711110 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:57.710971 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" event={"ID":"7112b666-6fe9-49f0-bc72-9433bedd4abb","Type":"ContainerStarted","Data":"d08dca64dc86c23849fd26b033bf05c79d1a7065de7877fa26a1dd4d8083e5b4"} Nov 01 01:04:57.827337 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:57.827225 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" podStartSLOduration=1.827207965 podStartE2EDuration="1.827207965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:04:57.826054737 +0000 UTC m=+2487.805637077" watchObservedRunningTime="2026-11-01 01:04:57.827207965 +0000 UTC m=+2487.806790299" Nov 01 01:04:58.720553 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:58.720524 4910 generic.go:356] "Generic (PLEG): container finished" podID="7112b666-6fe9-49f0-bc72-9433bedd4abb" containerID="35dd2d3a62cdd7746ce61bb6f2abc80fabe17b9fcf5a92d427add5f2526fb6f7" exitCode=0 Nov 01 01:04:58.720979 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:58.720606 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" event={"ID":"7112b666-6fe9-49f0-bc72-9433bedd4abb","Type":"ContainerDied","Data":"35dd2d3a62cdd7746ce61bb6f2abc80fabe17b9fcf5a92d427add5f2526fb6f7"} Nov 01 01:04:59.036035 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.036001 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b"] Nov 01 01:04:59.040808 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.040787 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.117260 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.117228 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.117460 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.117272 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhcb\" (UniqueName: \"kubernetes.io/projected/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-kube-api-access-vlhcb\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.117460 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.117431 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.125864 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.125833 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b"] Nov 01 01:04:59.219059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.218957 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.219219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.219092 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhcb\" (UniqueName: \"kubernetes.io/projected/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-kube-api-access-vlhcb\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.219219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.219158 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.219661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.219617 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.221759 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.221740 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.288490 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.288427 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhcb\" (UniqueName: \"kubernetes.io/projected/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-kube-api-access-vlhcb\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.355956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.355925 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:04:59.735106 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.735063 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b"] Nov 01 01:04:59.735410 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:04:59.735149 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4f6667_5eef_4f75_9fdb_3eab398c0b9b.slice/crio-f860248842b2e14e9b89bf1c51ec2b77fd535a54c47a1b24d3ce6760ecda8182 WatchSource:0}: Error finding container f860248842b2e14e9b89bf1c51ec2b77fd535a54c47a1b24d3ce6760ecda8182: Status 404 returned error can't find the container with id f860248842b2e14e9b89bf1c51ec2b77fd535a54c47a1b24d3ce6760ecda8182 Nov 01 01:04:59.844734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.840713 4910 patch_prober.go:29] interesting pod/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:04:59.844734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.840790 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:04:59.961493 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:04:59.961467 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:05:00.141678 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.141642 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-xks-ca\") pod \"7112b666-6fe9-49f0-bc72-9433bedd4abb\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " Nov 01 01:05:00.141882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.141688 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-openshift-ca\") pod \"7112b666-6fe9-49f0-bc72-9433bedd4abb\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " Nov 01 01:05:00.141882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.141742 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7112b666-6fe9-49f0-bc72-9433bedd4abb-kube-api-access-qf9jt\" (UniqueName: \"kubernetes.io/projected/7112b666-6fe9-49f0-bc72-9433bedd4abb-kube-api-access-qf9jt\") pod \"7112b666-6fe9-49f0-bc72-9433bedd4abb\" (UID: \"7112b666-6fe9-49f0-bc72-9433bedd4abb\") " Nov 01 01:05:00.142082 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.142055 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-openshift-ca" pod "7112b666-6fe9-49f0-bc72-9433bedd4abb" (UID: "7112b666-6fe9-49f0-bc72-9433bedd4abb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:00.142408 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.142347 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.143947 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.143916 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-xks-ca" pod "7112b666-6fe9-49f0-bc72-9433bedd4abb" (UID: "7112b666-6fe9-49f0-bc72-9433bedd4abb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:05:00.144099 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.144080 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7112b666-6fe9-49f0-bc72-9433bedd4abb-kube-api-access-qf9jt" pod "7112b666-6fe9-49f0-bc72-9433bedd4abb" (UID: "7112b666-6fe9-49f0-bc72-9433bedd4abb"). InnerVolumeSpecName "kube-api-access-qf9jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:00.243418 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.243387 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf9jt\" (UniqueName: \"kubernetes.io/projected/7112b666-6fe9-49f0-bc72-9433bedd4abb-kube-api-access-qf9jt\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.243547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.243424 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7112b666-6fe9-49f0-bc72-9433bedd4abb-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:00.740248 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.740216 4910 generic.go:356] "Generic (PLEG): container finished" podID="bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" containerID="0d15c88a5ffa84bc2d9da46631e66d1207db123294d47cb6e5e81250d23f4845" exitCode=0 Nov 01 01:05:00.740698 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.740297 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" event={"ID":"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b","Type":"ContainerDied","Data":"0d15c88a5ffa84bc2d9da46631e66d1207db123294d47cb6e5e81250d23f4845"} Nov 01 01:05:00.740698 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.740400 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" event={"ID":"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b","Type":"ContainerStarted","Data":"f860248842b2e14e9b89bf1c51ec2b77fd535a54c47a1b24d3ce6760ecda8182"} Nov 01 01:05:00.741924 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.741903 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" Nov 01 01:05:00.742024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.741928 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h" event={"ID":"7112b666-6fe9-49f0-bc72-9433bedd4abb","Type":"ContainerDied","Data":"d08dca64dc86c23849fd26b033bf05c79d1a7065de7877fa26a1dd4d8083e5b4"} Nov 01 01:05:00.742024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:00.741974 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08dca64dc86c23849fd26b033bf05c79d1a7065de7877fa26a1dd4d8083e5b4" Nov 01 01:05:01.925861 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:01.925832 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:05:02.065423 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.065340 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-xks-ca\") pod \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " Nov 01 01:05:02.065590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.065443 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-kube-api-access-vlhcb\" (UniqueName: \"kubernetes.io/projected/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-kube-api-access-vlhcb\") pod \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " Nov 01 01:05:02.065590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.065553 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-openshift-ca\") pod \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\" (UID: \"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b\") " Nov 01 01:05:02.066601 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.066570 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-openshift-ca" pod "bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" (UID: "bd4f6667-5eef-4f75-9fdb-3eab398c0b9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:02.081080 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.081047 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-kube-api-access-vlhcb" pod "bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" (UID: "bd4f6667-5eef-4f75-9fdb-3eab398c0b9b"). InnerVolumeSpecName "kube-api-access-vlhcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:02.082968 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.082937 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-xks-ca" pod "bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" (UID: "bd4f6667-5eef-4f75-9fdb-3eab398c0b9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:05:02.115376 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.115352 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:05:02.168077 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.168041 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.168077 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.168078 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.168347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.168094 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhcb\" (UniqueName: \"kubernetes.io/projected/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b-kube-api-access-vlhcb\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.268506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268476 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-data\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.268506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268509 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-credential-socket\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.268783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268558 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4749198e-b5c8-404f-88ca-f01bd68817a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4749198e-b5c8-404f-88ca-f01bd68817a7-istiod-ca-cert\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.268783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268590 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-podinfo\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.268783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268663 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-envoy\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.268783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268729 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-socket\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.268783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268779 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-token\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-token\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.269058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268849 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-data" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:02.269058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.268906 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-credential-socket" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:02.269058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269018 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-certs\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.269232 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269056 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-kube-api-access-pdcpl\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-kube-api-access-pdcpl\") pod \"4749198e-b5c8-404f-88ca-f01bd68817a7\" (UID: \"4749198e-b5c8-404f-88ca-f01bd68817a7\") " Nov 01 01:05:02.269232 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269089 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-socket" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:02.269340 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269280 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-certs" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:02.269737 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269711 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.269932 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269739 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.269932 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269744 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4749198e-b5c8-404f-88ca-f01bd68817a7-istiod-ca-cert" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:02.269932 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269753 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.269932 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.269791 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.271572 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.271542 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-podinfo" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:05:02.271946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.271921 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-token" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:02.272044 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.271993 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-kube-api-access-pdcpl" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "kube-api-access-pdcpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:02.273075 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.273054 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-envoy" pod "4749198e-b5c8-404f-88ca-f01bd68817a7" (UID: "4749198e-b5c8-404f-88ca-f01bd68817a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:02.371313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.371232 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcpl\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-kube-api-access-pdcpl\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.371313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.371257 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4749198e-b5c8-404f-88ca-f01bd68817a7-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.371313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.371267 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.371313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.371276 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.371313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.371286 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4749198e-b5c8-404f-88ca-f01bd68817a7-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:02.761172 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.761137 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" Nov 01 01:05:02.761172 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.761163 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b" event={"ID":"bd4f6667-5eef-4f75-9fdb-3eab398c0b9b","Type":"ContainerDied","Data":"f860248842b2e14e9b89bf1c51ec2b77fd535a54c47a1b24d3ce6760ecda8182"} Nov 01 01:05:02.761390 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.761210 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f860248842b2e14e9b89bf1c51ec2b77fd535a54c47a1b24d3ce6760ecda8182" Nov 01 01:05:02.763368 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.763342 4910 generic.go:356] "Generic (PLEG): container finished" podID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerID="0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c" exitCode=0 Nov 01 01:05:02.763516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.763407 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" Nov 01 01:05:02.763516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.763434 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" event={"ID":"4749198e-b5c8-404f-88ca-f01bd68817a7","Type":"ContainerDied","Data":"0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c"} Nov 01 01:05:02.763516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.763511 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd" event={"ID":"4749198e-b5c8-404f-88ca-f01bd68817a7","Type":"ContainerDied","Data":"1f86b2fd336d5f58bac3e4aff08185ba452170511bf9601c07073249882f8d11"} Nov 01 01:05:02.764247 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.763677 4910 scope.go:122] "RemoveContainer" containerID="0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c" Nov 01 01:05:02.777185 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.777162 4910 scope.go:122] "RemoveContainer" containerID="892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0" Nov 01 01:05:02.790872 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.790852 4910 scope.go:122] "RemoveContainer" containerID="0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c" Nov 01 01:05:02.791171 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:05:02.791154 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c\": container with ID starting with 0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c not found: ID does not exist" containerID="0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c" Nov 01 01:05:02.791236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.791180 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c"} err="failed to get container status \"0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c\": rpc error: code = NotFound desc = could not find container \"0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c\": container with ID starting with 0f0aa08a7686651e806d1c2c6cd08ed446b53a3e090861eb1433f1dfb1c0649c not found: ID does not exist" Nov 01 01:05:02.791236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.791199 4910 scope.go:122] "RemoveContainer" containerID="892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0" Nov 01 01:05:02.791418 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:05:02.791401 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0\": container with ID starting with 892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0 not found: ID does not exist" containerID="892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0" Nov 01 01:05:02.791461 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.791422 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0"} err="failed to get container status \"892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0\": rpc error: code = NotFound desc = could not find container \"892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0\": container with ID starting with 892bbe4220df365d1e8a9bc0cfcb7fc0cac13cca046abcd954f2c3257cc788e0 not found: ID does not exist" Nov 01 01:05:02.985822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:02.985786 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd"] Nov 01 01:05:03.036401 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:03.036321 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0f9d81-openshift-default-5f59bb9f9f-ffshd"] Nov 01 01:05:03.183665 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:03.183602 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4"] Nov 01 01:05:03.185670 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:03.185636 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" containerID="cri-o://a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9" gracePeriod=30 Nov 01 01:05:04.174053 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:04.174018 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:05:04.174506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:04.174102 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:05:04.194234 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:04.194203 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" path="/var/lib/kubelet/pods/4749198e-b5c8-404f-88ca-f01bd68817a7/volumes" Nov 01 01:05:08.466941 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.466878 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 01:05:08.485288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485257 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-data\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485304 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-certs\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485332 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/266e279e-d94f-481c-af70-1c0eb92163d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266e279e-d94f-481c-af70-1c0eb92163d4-istio-podinfo\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485358 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266e279e-d94f-481c-af70-1c0eb92163d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266e279e-d94f-481c-af70-1c0eb92163d4-istiod-ca-cert\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485387 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-istio-token\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-istio-token\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485443 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-kube-api-access-g4nnz\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-kube-api-access-g4nnz\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485502 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-credential-socket\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485551 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-data" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:08.485681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485565 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-envoy\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485928 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485676 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-certs" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:08.485928 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485744 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-socket\") pod \"266e279e-d94f-481c-af70-1c0eb92163d4\" (UID: \"266e279e-d94f-481c-af70-1c0eb92163d4\") " Nov 01 01:05:08.485928 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485870 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266e279e-d94f-481c-af70-1c0eb92163d4-istiod-ca-cert" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:08.485928 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485877 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-credential-socket" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:08.486131 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.485975 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-socket" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:08.486372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.486290 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.486372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.486317 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.486372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.486330 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266e279e-d94f-481c-af70-1c0eb92163d4-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.486372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.486343 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.486372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.486358 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.487990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.487957 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-kube-api-access-g4nnz" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "kube-api-access-g4nnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:08.487990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.487966 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/266e279e-d94f-481c-af70-1c0eb92163d4-istio-podinfo" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:05:08.487990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.487979 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-istio-token" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:08.488227 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.488076 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-envoy" pod "266e279e-d94f-481c-af70-1c0eb92163d4" (UID: "266e279e-d94f-481c-af70-1c0eb92163d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:08.587514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.587476 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4nnz\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-kube-api-access-g4nnz\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.587514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.587510 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266e279e-d94f-481c-af70-1c0eb92163d4-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.587514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.587521 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266e279e-d94f-481c-af70-1c0eb92163d4-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.587784 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.587531 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266e279e-d94f-481c-af70-1c0eb92163d4-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:08.857213 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.857172 4910 generic.go:356] "Generic (PLEG): container finished" podID="266e279e-d94f-481c-af70-1c0eb92163d4" containerID="a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9" exitCode=0 Nov 01 01:05:08.857403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.857273 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" Nov 01 01:05:08.857403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.857281 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" event={"ID":"266e279e-d94f-481c-af70-1c0eb92163d4","Type":"ContainerDied","Data":"a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9"} Nov 01 01:05:08.857403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.857340 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4" event={"ID":"266e279e-d94f-481c-af70-1c0eb92163d4","Type":"ContainerDied","Data":"1e6c026213634dee9bdda9d929ab28a5cf9de319071b847ea292aa72dd50118e"} Nov 01 01:05:08.857403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.857362 4910 scope.go:122] "RemoveContainer" containerID="a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9" Nov 01 01:05:08.869900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.869876 4910 scope.go:122] "RemoveContainer" containerID="1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669" Nov 01 01:05:08.880193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.880174 4910 scope.go:122] "RemoveContainer" containerID="a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9" Nov 01 01:05:08.880475 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:05:08.880457 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9\": container with ID starting with a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9 not found: ID does not exist" containerID="a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9" Nov 01 01:05:08.880527 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.880483 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9"} err="failed to get container status \"a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9\": rpc error: code = NotFound desc = could not find container \"a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9\": container with ID starting with a69e8a962011527d49b4007adee519ec4a2914545d2caca7aa46feb4f0dde1d9 not found: ID does not exist" Nov 01 01:05:08.880527 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.880502 4910 scope.go:122] "RemoveContainer" containerID="1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669" Nov 01 01:05:08.880799 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:05:08.880783 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669\": container with ID starting with 1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669 not found: ID does not exist" containerID="1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669" Nov 01 01:05:08.880851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:08.880814 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669"} err="failed to get container status \"1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669\": rpc error: code = NotFound desc = could not find container \"1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669\": container with ID starting with 1b95863857b3b760e08949c5e2052dd6cc714dfc778a210d05d556e6cd916669 not found: ID does not exist" Nov 01 01:05:09.053514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:09.050700 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4"] Nov 01 01:05:09.056507 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:09.056480 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e296f9-openshift-default-5b4867565-gnqt4"] Nov 01 01:05:10.195936 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:10.195901 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" path="/var/lib/kubelet/pods/266e279e-d94f-481c-af70-1c0eb92163d4/volumes" Nov 01 01:05:13.110079 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:05:13.110035 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:05:43.105644 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:05:43.105589 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:05:44.547262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:44.547229 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth"] Nov 01 01:05:44.547760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:44.547475 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" containerID="cri-o://af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627" gracePeriod=30 Nov 01 01:05:49.832262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.832205 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 01:05:49.959320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959292 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-socket\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959357 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-envoy\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959419 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28dbce44-5517-4767-9abf-d13454fa9148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbce44-5517-4767-9abf-d13454fa9148-istio-podinfo\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959476 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-certs\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959560 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-credential-socket\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959599 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28dbce44-5517-4767-9abf-d13454fa9148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbce44-5517-4767-9abf-d13454fa9148-istiod-ca-cert\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959616 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-data\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959616 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-socket" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:49.959852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959707 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-kube-api-access-xxhn9\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-kube-api-access-xxhn9\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.959852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959851 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-istio-token\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-istio-token\") pod \"28dbce44-5517-4767-9abf-d13454fa9148\" (UID: \"28dbce44-5517-4767-9abf-d13454fa9148\") " Nov 01 01:05:49.960183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959869 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-credential-socket" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:49.960183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.959911 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-certs" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:49.960183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.960021 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-data" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:49.960183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.960111 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dbce44-5517-4767-9abf-d13454fa9148-istiod-ca-cert" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:05:49.960417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.960383 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:49.960417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.960402 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbce44-5517-4767-9abf-d13454fa9148-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:49.960417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.960415 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:49.960590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.960430 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:49.960590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.960444 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:49.962383 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.962341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28dbce44-5517-4767-9abf-d13454fa9148-istio-podinfo" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:05:49.962596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.962574 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-istio-token" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:49.962721 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.962681 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-envoy" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:05:49.962789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:49.962765 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-kube-api-access-xxhn9" pod "28dbce44-5517-4767-9abf-d13454fa9148" (UID: "28dbce44-5517-4767-9abf-d13454fa9148"). InnerVolumeSpecName "kube-api-access-xxhn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:05:50.061666 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.061601 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhn9\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-kube-api-access-xxhn9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:50.061666 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.061668 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28dbce44-5517-4767-9abf-d13454fa9148-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:50.061890 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.061686 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbce44-5517-4767-9abf-d13454fa9148-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:50.061890 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.061699 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbce44-5517-4767-9abf-d13454fa9148-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:05:50.316773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.316737 4910 generic.go:356] "Generic (PLEG): container finished" podID="28dbce44-5517-4767-9abf-d13454fa9148" containerID="af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627" exitCode=0 Nov 01 01:05:50.316955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.316793 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" Nov 01 01:05:50.316955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.316792 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" event={"ID":"28dbce44-5517-4767-9abf-d13454fa9148","Type":"ContainerDied","Data":"af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627"} Nov 01 01:05:50.316955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.316854 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth" event={"ID":"28dbce44-5517-4767-9abf-d13454fa9148","Type":"ContainerDied","Data":"c2633b9c104df8fce0aaf400ff3e52f4a1dc020d037033efd7bc1d69d37dd824"} Nov 01 01:05:50.316955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.316876 4910 scope.go:122] "RemoveContainer" containerID="af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627" Nov 01 01:05:50.330323 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.330288 4910 scope.go:122] "RemoveContainer" containerID="af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627" Nov 01 01:05:50.330700 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:05:50.330676 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627\": container with ID starting with af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627 not found: ID does not exist" containerID="af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627" Nov 01 01:05:50.330831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.330706 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627"} err="failed to get container status \"af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627\": rpc error: code = NotFound desc = could not find container \"af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627\": container with ID starting with af06a76e7d527378ece4d0eecd0c187c889f497ab99b5178c1d1f1e6f4a54627 not found: ID does not exist" Nov 01 01:05:50.485545 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.485439 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth"] Nov 01 01:05:50.493898 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:50.493866 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d4mth"] Nov 01 01:05:52.196253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:52.196218 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dbce44-5517-4767-9abf-d13454fa9148" path="/var/lib/kubelet/pods/28dbce44-5517-4767-9abf-d13454fa9148/volumes" Nov 01 01:05:53.240171 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:53.240136 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b"] Nov 01 01:05:53.290057 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:53.290021 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-99eb0f-82b9b"] Nov 01 01:05:54.195066 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:05:54.195026 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" path="/var/lib/kubelet/pods/bd4f6667-5eef-4f75-9fdb-3eab398c0b9b/volumes" Nov 01 01:06:00.010091 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.010036 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26"] Nov 01 01:06:00.010996 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.010960 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" containerID="cri-o://61ec14e801f61940fbda1d98074d0c97df8dc64e42d87e42cdff4bee8475a3a8" gracePeriod=30 Nov 01 01:06:00.278953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.278871 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:06:00.278953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.278920 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:06:00.878582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.878526 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv"] Nov 01 01:06:00.879423 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879398 4910 cpu_manager.go:414] "containerMap: removing container" podUID="bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" containerName="revoke-keys" Nov 01 01:06:00.879423 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879423 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" containerName="revoke-keys" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879440 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879448 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879477 4910 cpu_manager.go:414] "containerMap: removing container" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879485 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879506 4910 cpu_manager.go:414] "containerMap: removing container" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879514 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879542 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7112b666-6fe9-49f0-bc72-9433bedd4abb" containerName="revoke-keys" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879549 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7112b666-6fe9-49f0-bc72-9433bedd4abb" containerName="revoke-keys" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879585 4910 cpu_manager.go:414] "containerMap: removing container" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" Nov 01 01:06:00.879662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879594 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" Nov 01 01:06:00.880201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879868 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4f6667-5eef-4f75-9fdb-3eab398c0b9b" containerName="revoke-keys" Nov 01 01:06:00.880201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879903 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" Nov 01 01:06:00.880201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879918 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dbce44-5517-4767-9abf-d13454fa9148" containerName="istio-proxy" Nov 01 01:06:00.880201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879933 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" Nov 01 01:06:00.880201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879945 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7112b666-6fe9-49f0-bc72-9433bedd4abb" containerName="revoke-keys" Nov 01 01:06:00.880201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.879960 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="266e279e-d94f-481c-af70-1c0eb92163d4" containerName="istio-proxy" Nov 01 01:06:00.884193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.884172 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.894704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.894683 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.894895 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.894873 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftksb\" (UniqueName: \"kubernetes.io/projected/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-kube-api-access-ftksb\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.895007 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.894993 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.936876 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.936819 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:06:00.937022 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.937005 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:06:00.939591 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.939565 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv"] Nov 01 01:06:00.996204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.996176 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.996370 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.996305 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftksb\" (UniqueName: \"kubernetes.io/projected/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-kube-api-access-ftksb\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.996417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.996389 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.997042 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.997007 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:00.999330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:00.999288 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:01.079079 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:01.079044 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftksb\" (UniqueName: \"kubernetes.io/projected/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-kube-api-access-ftksb\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:01.199196 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:01.199115 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:01.632122 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:01.632082 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv"] Nov 01 01:06:01.648777 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:06:01.648744 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36b4faf_9322_41c6_bf9c_8f6af66f9c04.slice/crio-411c83a2ad22a37575c0d66183b281b5d10fc8ecfab76e68c2c6b75a1b45b2ae WatchSource:0}: Error finding container 411c83a2ad22a37575c0d66183b281b5d10fc8ecfab76e68c2c6b75a1b45b2ae: Status 404 returned error can't find the container with id 411c83a2ad22a37575c0d66183b281b5d10fc8ecfab76e68c2c6b75a1b45b2ae Nov 01 01:06:02.424297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:02.424185 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" event={"ID":"f36b4faf-9322-41c6-bf9c-8f6af66f9c04","Type":"ContainerStarted","Data":"284bfc389881e84a18e5afad2055506367bd18312a0bfa2ea337e1f7d2cd0423"} Nov 01 01:06:02.424297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:02.424252 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" event={"ID":"f36b4faf-9322-41c6-bf9c-8f6af66f9c04","Type":"ContainerStarted","Data":"411c83a2ad22a37575c0d66183b281b5d10fc8ecfab76e68c2c6b75a1b45b2ae"} Nov 01 01:06:02.540426 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:02.540343 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" podStartSLOduration=2.540328928 podStartE2EDuration="2.540328928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:02.53979925 +0000 UTC m=+2552.519381643" watchObservedRunningTime="2026-11-01 01:06:02.540328928 +0000 UTC m=+2552.519911262" Nov 01 01:06:03.435368 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:03.435333 4910 generic.go:356] "Generic (PLEG): container finished" podID="f36b4faf-9322-41c6-bf9c-8f6af66f9c04" containerID="284bfc389881e84a18e5afad2055506367bd18312a0bfa2ea337e1f7d2cd0423" exitCode=0 Nov 01 01:06:03.436001 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:03.435421 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" event={"ID":"f36b4faf-9322-41c6-bf9c-8f6af66f9c04","Type":"ContainerDied","Data":"284bfc389881e84a18e5afad2055506367bd18312a0bfa2ea337e1f7d2cd0423"} Nov 01 01:06:05.006495 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.006456 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:05.151174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.149544 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-kube-api-access-ftksb\" (UniqueName: \"kubernetes.io/projected/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-kube-api-access-ftksb\") pod \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " Nov 01 01:06:05.151174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.149765 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-xks-ca\") pod \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " Nov 01 01:06:05.151174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.149922 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-openshift-ca\") pod \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\" (UID: \"f36b4faf-9322-41c6-bf9c-8f6af66f9c04\") " Nov 01 01:06:05.152494 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.151840 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-openshift-ca" pod "f36b4faf-9322-41c6-bf9c-8f6af66f9c04" (UID: "f36b4faf-9322-41c6-bf9c-8f6af66f9c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:05.166054 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.165984 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-kube-api-access-ftksb" pod "f36b4faf-9322-41c6-bf9c-8f6af66f9c04" (UID: "f36b4faf-9322-41c6-bf9c-8f6af66f9c04"). InnerVolumeSpecName "kube-api-access-ftksb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:05.195732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.195375 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-xks-ca" pod "f36b4faf-9322-41c6-bf9c-8f6af66f9c04" (UID: "f36b4faf-9322-41c6-bf9c-8f6af66f9c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:06:05.262723 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.257733 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftksb\" (UniqueName: \"kubernetes.io/projected/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-kube-api-access-ftksb\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.262723 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.257769 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.262723 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.257788 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b4faf-9322-41c6-bf9c-8f6af66f9c04-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.440301 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.439825 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh"] Nov 01 01:06:05.444289 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.443838 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72acf4-62kgh"] Nov 01 01:06:05.512664 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.512275 4910 generic.go:356] "Generic (PLEG): container finished" podID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerID="61ec14e801f61940fbda1d98074d0c97df8dc64e42d87e42cdff4bee8475a3a8" exitCode=0 Nov 01 01:06:05.512664 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.512466 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" event={"ID":"14e820b1-53a8-446d-a90a-20b32e9e08e6","Type":"ContainerDied","Data":"61ec14e801f61940fbda1d98074d0c97df8dc64e42d87e42cdff4bee8475a3a8"} Nov 01 01:06:05.512664 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.512511 4910 scope.go:122] "RemoveContainer" containerID="799090d7e859b22b90564ea08e1ee7210f6819e79d38ae39fb6a42fab6782dae" Nov 01 01:06:05.516598 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.515439 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" Nov 01 01:06:05.516598 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.515552 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv" event={"ID":"f36b4faf-9322-41c6-bf9c-8f6af66f9c04","Type":"ContainerDied","Data":"411c83a2ad22a37575c0d66183b281b5d10fc8ecfab76e68c2c6b75a1b45b2ae"} Nov 01 01:06:05.516598 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.515685 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411c83a2ad22a37575c0d66183b281b5d10fc8ecfab76e68c2c6b75a1b45b2ae" Nov 01 01:06:05.535221 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.535127 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 01:06:05.688192 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688157 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-envoy\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688239 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-kube-api-access-q5j7s\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-kube-api-access-q5j7s\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688321 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-socket\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688450 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688386 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-credential-socket\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688479 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-data\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688520 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-certs\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688559 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e820b1-53a8-446d-a90a-20b32e9e08e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e820b1-53a8-446d-a90a-20b32e9e08e6-istiod-ca-cert\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688658 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-podinfo\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.688688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688685 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-token\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-token\") pod \"14e820b1-53a8-446d-a90a-20b32e9e08e6\" (UID: \"14e820b1-53a8-446d-a90a-20b32e9e08e6\") " Nov 01 01:06:05.689009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688718 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-socket" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:05.689009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.688894 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-certs" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:05.689851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.689355 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.689851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.689382 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.689851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.689668 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e820b1-53a8-446d-a90a-20b32e9e08e6-istiod-ca-cert" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:05.689851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.689822 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-data" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:05.690484 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.690417 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-credential-socket" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:05.691264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.691240 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-envoy" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:05.691264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.691256 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-token" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:05.692217 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.692195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-kube-api-access-q5j7s" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "kube-api-access-q5j7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:05.692944 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.692912 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-podinfo" pod "14e820b1-53a8-446d-a90a-20b32e9e08e6" (UID: "14e820b1-53a8-446d-a90a-20b32e9e08e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:06:05.791204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.791122 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.791400 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.791385 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e820b1-53a8-446d-a90a-20b32e9e08e6-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.791468 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.791458 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.791535 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.791528 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.791588 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.791581 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.791661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.791652 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5j7s\" (UniqueName: \"kubernetes.io/projected/14e820b1-53a8-446d-a90a-20b32e9e08e6-kube-api-access-q5j7s\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:05.791726 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:05.791718 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e820b1-53a8-446d-a90a-20b32e9e08e6-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:06.193329 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.193235 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f777449-8ed0-419e-afd6-a3ffce290448" path="/var/lib/kubelet/pods/8f777449-8ed0-419e-afd6-a3ffce290448/volumes" Nov 01 01:06:06.527950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.527918 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" Nov 01 01:06:06.528181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.527947 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26" event={"ID":"14e820b1-53a8-446d-a90a-20b32e9e08e6","Type":"ContainerDied","Data":"2e4f7ef1d280985ed8f3276eeb63da8ba4315ab7fe9f285bb2293f76c46bd8c9"} Nov 01 01:06:06.528181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.528023 4910 scope.go:122] "RemoveContainer" containerID="61ec14e801f61940fbda1d98074d0c97df8dc64e42d87e42cdff4bee8475a3a8" Nov 01 01:06:06.635373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.635340 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h"] Nov 01 01:06:06.639476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.639448 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-de2a95-7mp7h"] Nov 01 01:06:06.766741 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.766704 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26"] Nov 01 01:06:06.777087 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:06.777044 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99eb0f-openshift-default-d788689f-wtx26"] Nov 01 01:06:08.193778 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:08.193744 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" path="/var/lib/kubelet/pods/14e820b1-53a8-446d-a90a-20b32e9e08e6/volumes" Nov 01 01:06:08.194303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:08.194287 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7112b666-6fe9-49f0-bc72-9433bedd4abb" path="/var/lib/kubelet/pods/7112b666-6fe9-49f0-bc72-9433bedd4abb/volumes" Nov 01 01:06:12.346936 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:12.346901 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq"] Nov 01 01:06:12.347381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:12.347113 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" containerID="cri-o://976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e" gracePeriod=30 Nov 01 01:06:13.110605 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:06:13.110568 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:06:14.512015 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.511959 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc"] Nov 01 01:06:14.512809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.512778 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" containerID="cri-o://9cd4661b5fef8a5c8e5c54814333cf68fc4338cc8064a24948c094b2ec3c24fa" gracePeriod=30 Nov 01 01:06:14.826212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826138 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz"] Nov 01 01:06:14.826743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826729 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f36b4faf-9322-41c6-bf9c-8f6af66f9c04" containerName="revoke-keys" Nov 01 01:06:14.826802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826744 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36b4faf-9322-41c6-bf9c-8f6af66f9c04" containerName="revoke-keys" Nov 01 01:06:14.826802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826755 4910 cpu_manager.go:414] "containerMap: removing container" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" Nov 01 01:06:14.826802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826760 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" Nov 01 01:06:14.826802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826780 4910 cpu_manager.go:414] "containerMap: removing container" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" Nov 01 01:06:14.826802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826785 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" Nov 01 01:06:14.826802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826795 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" Nov 01 01:06:14.826802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826800 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" Nov 01 01:06:14.827026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826879 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36b4faf-9322-41c6-bf9c-8f6af66f9c04" containerName="revoke-keys" Nov 01 01:06:14.827026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826887 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" Nov 01 01:06:14.827026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826897 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e820b1-53a8-446d-a90a-20b32e9e08e6" containerName="istio-proxy" Nov 01 01:06:14.827026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.826903 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4749198e-b5c8-404f-88ca-f01bd68817a7" containerName="istio-proxy" Nov 01 01:06:14.831963 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.831943 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:14.886408 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.886377 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:06:14.886748 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.886711 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:06:14.888424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.888388 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz"] Nov 01 01:06:14.924776 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.924738 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:14.924962 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.924859 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jfqn\" (UniqueName: \"kubernetes.io/projected/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-kube-api-access-2jfqn\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:14.925038 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:14.924974 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.025903 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.025868 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.026092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.026031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.026164 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.026105 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jfqn\" (UniqueName: \"kubernetes.io/projected/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-kube-api-access-2jfqn\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.026719 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.026692 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.029256 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.029229 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.087821 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.087737 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jfqn\" (UniqueName: \"kubernetes.io/projected/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-kube-api-access-2jfqn\") pod \"maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.157842 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.157801 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:15.541998 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.541973 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz"] Nov 01 01:06:15.542488 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:06:15.542462 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0438b1_c83c_42d1_87f2_b68235f1ef2d.slice/crio-784d8cdb255a96eab8a3d2bef78df95ff227f5865451ea51ca2589757b6c48ad WatchSource:0}: Error finding container 784d8cdb255a96eab8a3d2bef78df95ff227f5865451ea51ca2589757b6c48ad: Status 404 returned error can't find the container with id 784d8cdb255a96eab8a3d2bef78df95ff227f5865451ea51ca2589757b6c48ad Nov 01 01:06:15.685412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:15.685367 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" event={"ID":"ae0438b1-c83c-42d1-87f2-b68235f1ef2d","Type":"ContainerStarted","Data":"784d8cdb255a96eab8a3d2bef78df95ff227f5865451ea51ca2589757b6c48ad"} Nov 01 01:06:16.696043 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:16.696012 4910 generic.go:356] "Generic (PLEG): container finished" podID="ae0438b1-c83c-42d1-87f2-b68235f1ef2d" containerID="3d03f945aaa7d4b1a5669da9713b146ce72596bf1ba3623fce31ea8cf05a0a71" exitCode=0 Nov 01 01:06:16.696457 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:16.696085 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" event={"ID":"ae0438b1-c83c-42d1-87f2-b68235f1ef2d","Type":"ContainerDied","Data":"3d03f945aaa7d4b1a5669da9713b146ce72596bf1ba3623fce31ea8cf05a0a71"} Nov 01 01:06:17.654589 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.654562 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:06:17.673831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.673791 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-data\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.673993 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.673870 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-envoy\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.673993 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.673901 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-credential-socket\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.673993 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.673926 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-socket\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.674208 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674102 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-podinfo\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.674208 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674146 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-data" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:17.674208 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674189 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-certs\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.674438 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674312 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54137c0-9741-4bf6-8c56-a57244f6d7db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54137c0-9741-4bf6-8c56-a57244f6d7db-istiod-ca-cert\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.674438 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674361 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-token\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-token\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.674438 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674394 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-kube-api-access-hj7hf\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-kube-api-access-hj7hf\") pod \"d54137c0-9741-4bf6-8c56-a57244f6d7db\" (UID: \"d54137c0-9741-4bf6-8c56-a57244f6d7db\") " Nov 01 01:06:17.674438 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674404 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-certs" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:17.674719 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674683 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-socket" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:17.674719 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674710 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54137c0-9741-4bf6-8c56-a57244f6d7db-istiod-ca-cert" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:17.675196 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.674921 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-credential-socket" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:17.675577 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.675556 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.675709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.675583 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54137c0-9741-4bf6-8c56-a57244f6d7db-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.675709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.675600 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.675709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.675615 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.675709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.675668 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.676893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.676866 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-podinfo" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:06:17.677006 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.676872 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-envoy" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:17.677212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.677195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-token" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:17.677487 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.677462 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-kube-api-access-hj7hf" pod "d54137c0-9741-4bf6-8c56-a57244f6d7db" (UID: "d54137c0-9741-4bf6-8c56-a57244f6d7db"). InnerVolumeSpecName "kube-api-access-hj7hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:17.706785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.706745 4910 generic.go:356] "Generic (PLEG): container finished" podID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerID="976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e" exitCode=0 Nov 01 01:06:17.707238 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.706797 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" Nov 01 01:06:17.707238 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.706830 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" event={"ID":"d54137c0-9741-4bf6-8c56-a57244f6d7db","Type":"ContainerDied","Data":"976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e"} Nov 01 01:06:17.707238 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.706890 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq" event={"ID":"d54137c0-9741-4bf6-8c56-a57244f6d7db","Type":"ContainerDied","Data":"238afc1edd17baa41db52400fc891575ec0473a85fba01eb2862bab1206782b2"} Nov 01 01:06:17.707238 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.706920 4910 scope.go:122] "RemoveContainer" containerID="976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e" Nov 01 01:06:17.722733 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.722713 4910 scope.go:122] "RemoveContainer" containerID="e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579" Nov 01 01:06:17.743268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.743161 4910 scope.go:122] "RemoveContainer" containerID="976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e" Nov 01 01:06:17.744000 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:06:17.743849 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e\": container with ID starting with 976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e not found: ID does not exist" containerID="976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e" Nov 01 01:06:17.744000 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.743898 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e"} err="failed to get container status \"976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e\": rpc error: code = NotFound desc = could not find container \"976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e\": container with ID starting with 976808416b121e80fb5c39b841c0ac65db4d382c106fd5daf10b799589243d9e not found: ID does not exist" Nov 01 01:06:17.744000 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.743922 4910 scope.go:122] "RemoveContainer" containerID="e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579" Nov 01 01:06:17.745150 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:06:17.745120 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579\": container with ID starting with e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579 not found: ID does not exist" containerID="e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579" Nov 01 01:06:17.745282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.745158 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579"} err="failed to get container status \"e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579\": rpc error: code = NotFound desc = could not find container \"e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579\": container with ID starting with e3a67d56e387a0aff253c99f913dd6daa912a5ebd54716934a4ddf08c15b4579 not found: ID does not exist" Nov 01 01:06:17.777668 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.777245 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.777668 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.777280 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.777668 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.777299 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7hf\" (UniqueName: \"kubernetes.io/projected/d54137c0-9741-4bf6-8c56-a57244f6d7db-kube-api-access-hj7hf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.777668 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.777313 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54137c0-9741-4bf6-8c56-a57244f6d7db-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.862230 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.862198 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:17.878482 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.878419 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-kube-api-access-2jfqn\" (UniqueName: \"kubernetes.io/projected/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-kube-api-access-2jfqn\") pod \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " Nov 01 01:06:17.878482 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.878470 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-openshift-ca\") pod \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " Nov 01 01:06:17.878715 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.878673 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-xks-ca\") pod \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\" (UID: \"ae0438b1-c83c-42d1-87f2-b68235f1ef2d\") " Nov 01 01:06:17.878914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.878883 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-openshift-ca" pod "ae0438b1-c83c-42d1-87f2-b68235f1ef2d" (UID: "ae0438b1-c83c-42d1-87f2-b68235f1ef2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:17.879705 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.879679 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.881081 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.881054 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-kube-api-access-2jfqn" pod "ae0438b1-c83c-42d1-87f2-b68235f1ef2d" (UID: "ae0438b1-c83c-42d1-87f2-b68235f1ef2d"). InnerVolumeSpecName "kube-api-access-2jfqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:17.881178 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.881051 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-xks-ca" pod "ae0438b1-c83c-42d1-87f2-b68235f1ef2d" (UID: "ae0438b1-c83c-42d1-87f2-b68235f1ef2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:06:17.894264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.894235 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq"] Nov 01 01:06:17.898156 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.898134 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-72acf4-openshift-default-669ddf676f-bwjpq"] Nov 01 01:06:17.980795 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.980762 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jfqn\" (UniqueName: \"kubernetes.io/projected/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-kube-api-access-2jfqn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:17.980795 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:17.980789 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0438b1-c83c-42d1-87f2-b68235f1ef2d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:18.194103 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:18.194012 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" path="/var/lib/kubelet/pods/d54137c0-9741-4bf6-8c56-a57244f6d7db/volumes" Nov 01 01:06:18.727344 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:18.727317 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" Nov 01 01:06:18.727854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:18.727309 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz" event={"ID":"ae0438b1-c83c-42d1-87f2-b68235f1ef2d","Type":"ContainerDied","Data":"784d8cdb255a96eab8a3d2bef78df95ff227f5865451ea51ca2589757b6c48ad"} Nov 01 01:06:18.727854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:18.727468 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="784d8cdb255a96eab8a3d2bef78df95ff227f5865451ea51ca2589757b6c48ad" Nov 01 01:06:19.743061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.743024 4910 generic.go:356] "Generic (PLEG): container finished" podID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerID="9cd4661b5fef8a5c8e5c54814333cf68fc4338cc8064a24948c094b2ec3c24fa" exitCode=0 Nov 01 01:06:19.743577 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.743089 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" event={"ID":"a3c237a6-a5ed-488c-85d5-fa0ea4993c65","Type":"ContainerDied","Data":"9cd4661b5fef8a5c8e5c54814333cf68fc4338cc8064a24948c094b2ec3c24fa"} Nov 01 01:06:19.743577 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.743160 4910 scope.go:122] "RemoveContainer" containerID="1fc1fe192823f61332071797fc0848d3933f2f3324f7c856e5b7cfa506a78730" Nov 01 01:06:19.826177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.826153 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:06:19.911388 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911358 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-data\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911445 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-socket\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911472 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-envoy\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911540 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istiod-ca-cert\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911562 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-credential-socket\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911582 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-podinfo\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911680 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-certs\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911715 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-data" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:19.911950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911764 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-kube-api-access-l8s5q\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-kube-api-access-l8s5q\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911782 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-socket" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:19.911950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911819 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-token\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-token\") pod \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\" (UID: \"a3c237a6-a5ed-488c-85d5-fa0ea4993c65\") " Nov 01 01:06:19.911950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.911867 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-credential-socket" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:19.912281 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.912010 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-certs" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:19.912281 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.912023 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istiod-ca-cert" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:19.912564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.912306 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:19.912564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.912322 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:19.912564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.912331 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:19.912564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.912341 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:19.912564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.912353 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:19.914097 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.914076 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-podinfo" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:06:19.914207 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.914109 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-kube-api-access-l8s5q" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "kube-api-access-l8s5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:19.914505 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.914473 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-envoy" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:19.914505 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:19.914489 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-token" pod "a3c237a6-a5ed-488c-85d5-fa0ea4993c65" (UID: "a3c237a6-a5ed-488c-85d5-fa0ea4993c65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:20.014787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.014022 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:20.014787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.014080 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:20.014787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.014096 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8s5q\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-kube-api-access-l8s5q\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:20.014787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.014110 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c237a6-a5ed-488c-85d5-fa0ea4993c65-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:20.753415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.753372 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" event={"ID":"a3c237a6-a5ed-488c-85d5-fa0ea4993c65","Type":"ContainerDied","Data":"9564e37f46bb9886bd891e53953711def87f0ac67d312bd1fdd2763fbad86562"} Nov 01 01:06:20.753415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.753395 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc" Nov 01 01:06:20.753977 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.753445 4910 scope.go:122] "RemoveContainer" containerID="9cd4661b5fef8a5c8e5c54814333cf68fc4338cc8064a24948c094b2ec3c24fa" Nov 01 01:06:20.974038 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:20.974008 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc"] Nov 01 01:06:21.023063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:21.022989 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-de2a95-openshift-default-d87868ddf-m28xc"] Nov 01 01:06:22.194417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:22.194374 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" path="/var/lib/kubelet/pods/a3c237a6-a5ed-488c-85d5-fa0ea4993c65/volumes" Nov 01 01:06:26.680306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:26.680270 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq"] Nov 01 01:06:31.680854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:31.680816 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" podUID="56a036e7-5a71-4ff4-9a77-bc23f2249437" containerName="payload-processing" containerID="cri-o://cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603" gracePeriod=25 Nov 01 01:06:33.611791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:33.611744 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" podUID="56a036e7-5a71-4ff4-9a77-bc23f2249437" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:06:34.953181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:34.953141 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc"] Nov 01 01:06:35.000919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:35.000884 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f97e89-zfdsc"] Nov 01 01:06:36.195582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:36.195540 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee312436-e2ce-4946-bf21-a8914942ed47" path="/var/lib/kubelet/pods/ee312436-e2ce-4946-bf21-a8914942ed47/volumes" Nov 01 01:06:38.662042 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662009 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc"] Nov 01 01:06:38.662585 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662570 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" Nov 01 01:06:38.662687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662587 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" Nov 01 01:06:38.662687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662642 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" Nov 01 01:06:38.662687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662650 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" Nov 01 01:06:38.662687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662665 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" Nov 01 01:06:38.662687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662671 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" Nov 01 01:06:38.662687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662680 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ae0438b1-c83c-42d1-87f2-b68235f1ef2d" containerName="revoke-keys" Nov 01 01:06:38.662687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662684 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0438b1-c83c-42d1-87f2-b68235f1ef2d" containerName="revoke-keys" Nov 01 01:06:38.663072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662775 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" Nov 01 01:06:38.663072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662788 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" Nov 01 01:06:38.663072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662796 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" Nov 01 01:06:38.663072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662802 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c237a6-a5ed-488c-85d5-fa0ea4993c65" containerName="istio-proxy" Nov 01 01:06:38.663072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.662807 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0438b1-c83c-42d1-87f2-b68235f1ef2d" containerName="revoke-keys" Nov 01 01:06:38.668128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.668106 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830109 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830080 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2db\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-kube-api-access-gw2db\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830316 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830122 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830316 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830224 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf97496-8a64-4a79-8f7c-5135cae1279d-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830316 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830276 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830505 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830469 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830505 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830498 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830605 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830528 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830605 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830597 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.830718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.830668 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.881934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.881903 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc"] Nov 01 01:06:38.931689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931579 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.931689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931609 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.931689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931644 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.931689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931670 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931720 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931752 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2db\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-kube-api-access-gw2db\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931783 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931847 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf97496-8a64-4a79-8f7c-5135cae1279d-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.931880 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.932006 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932446 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.932100 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932446 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.932156 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932446 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.932210 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.932768 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.932701 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf97496-8a64-4a79-8f7c-5135cae1279d-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.934307 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.934279 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.934407 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.934389 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.993816 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.993776 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:38.994221 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:38.994197 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2db\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-kube-api-access-gw2db\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:39.281870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:39.281836 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:39.731376 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:39.730876 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc"] Nov 01 01:06:39.732969 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:06:39.732938 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf97496_8a64_4a79_8f7c_5135cae1279d.slice/crio-cf6be70ed2df7d9a5f4907551c1b19eff5ab0afef48a75347e1f36e266e319d5 WatchSource:0}: Error finding container cf6be70ed2df7d9a5f4907551c1b19eff5ab0afef48a75347e1f36e266e319d5: Status 404 returned error can't find the container with id cf6be70ed2df7d9a5f4907551c1b19eff5ab0afef48a75347e1f36e266e319d5 Nov 01 01:06:39.735966 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:39.735925 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.736069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:39.736013 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:39.736069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:39.736054 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:40.007547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:40.007504 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" event={"ID":"0bf97496-8a64-4a79-8f7c-5135cae1279d","Type":"ContainerStarted","Data":"62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e"} Nov 01 01:06:40.007785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:40.007566 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" event={"ID":"0bf97496-8a64-4a79-8f7c-5135cae1279d","Type":"ContainerStarted","Data":"cf6be70ed2df7d9a5f4907551c1b19eff5ab0afef48a75347e1f36e266e319d5"} Nov 01 01:06:40.136928 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:40.136874 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podStartSLOduration=2.136860673 podStartE2EDuration="2.136860673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:40.134389729 +0000 UTC m=+2590.113972064" watchObservedRunningTime="2026-11-01 01:06:40.136860673 +0000 UTC m=+2590.116443079" Nov 01 01:06:40.282351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:40.282270 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:40.283837 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:40.283810 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 01:06:40.284005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:40.283867 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 01:06:41.284522 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:41.284483 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 01:06:41.285102 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:41.284581 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 01:06:42.004939 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.004897 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l"] Nov 01 01:06:42.005843 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.005802 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" containerID="cri-o://b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93" gracePeriod=30 Nov 01 01:06:42.284303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.284213 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 01:06:42.284571 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.284544 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 01:06:42.822074 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.822039 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr"] Nov 01 01:06:42.822722 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.822706 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" Nov 01 01:06:42.822784 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.822723 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54137c0-9741-4bf6-8c56-a57244f6d7db" containerName="istio-proxy" Nov 01 01:06:42.826497 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.826480 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:42.891003 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.890973 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:06:42.891281 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.891266 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:06:42.898601 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.896905 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr"] Nov 01 01:06:42.932266 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.932234 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:42.932440 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.932268 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:42.932440 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:42.932370 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrhcw\" (UniqueName: \"kubernetes.io/projected/9235253a-2fac-4fa5-89ef-f478c11c5fca-kube-api-access-nrhcw\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.033930 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.033896 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.034111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.033938 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.034111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.033975 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhcw\" (UniqueName: \"kubernetes.io/projected/9235253a-2fac-4fa5-89ef-f478c11c5fca-kube-api-access-nrhcw\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.034826 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.034762 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.037179 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.037154 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.096844 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.096767 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrhcw\" (UniqueName: \"kubernetes.io/projected/9235253a-2fac-4fa5-89ef-f478c11c5fca-kube-api-access-nrhcw\") pod \"maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.108837 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:06:43.108803 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:06:43.141529 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.141504 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:43.285333 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.285292 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 01:06:43.285921 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.285357 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 01:06:43.308703 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.306044 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92"] Nov 01 01:06:43.312769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.312387 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.338273 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.338240 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc77c\" (UniqueName: \"kubernetes.io/projected/6d157165-8ae9-465a-ad2c-6c647bd01025-kube-api-access-zc77c\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.338416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.338299 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.338416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.338386 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.370822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.370350 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92"] Nov 01 01:06:43.439262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.439228 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc77c\" (UniqueName: \"kubernetes.io/projected/6d157165-8ae9-465a-ad2c-6c647bd01025-kube-api-access-zc77c\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.439453 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.439274 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.439453 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.439342 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.439924 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.439901 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.441875 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.441852 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.500708 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.500676 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc77c\" (UniqueName: \"kubernetes.io/projected/6d157165-8ae9-465a-ad2c-6c647bd01025-kube-api-access-zc77c\") pod \"maas-api-revoke-keys-e2e-mt-b112c158-7lh92\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:43.578651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.577749 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr"] Nov 01 01:06:43.578651 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:06:43.578534 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9235253a_2fac_4fa5_89ef_f478c11c5fca.slice/crio-f3d695d34292feffce54656c1833e76760ae98516034ed6885fd50225f2f4c80 WatchSource:0}: Error finding container f3d695d34292feffce54656c1833e76760ae98516034ed6885fd50225f2f4c80: Status 404 returned error can't find the container with id f3d695d34292feffce54656c1833e76760ae98516034ed6885fd50225f2f4c80 Nov 01 01:06:43.612868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.612840 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" podUID="56a036e7-5a71-4ff4-9a77-bc23f2249437" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:06:43.628905 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:43.628697 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:44.072205 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.070951 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" event={"ID":"9235253a-2fac-4fa5-89ef-f478c11c5fca","Type":"ContainerStarted","Data":"fa28c15a18b46c2c721fea2aeb6cf8fb4c7b32e57869c772165ff445ee69a769"} Nov 01 01:06:44.072205 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.071066 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" event={"ID":"9235253a-2fac-4fa5-89ef-f478c11c5fca","Type":"ContainerStarted","Data":"f3d695d34292feffce54656c1833e76760ae98516034ed6885fd50225f2f4c80"} Nov 01 01:06:44.178646 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.178297 4910 patch_prober.go:29] interesting pod/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:06:44.178646 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.178353 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:06:44.192486 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.192428 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" podStartSLOduration=2.192413664 podStartE2EDuration="2.192413664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:44.187504886 +0000 UTC m=+2594.167087244" watchObservedRunningTime="2026-11-01 01:06:44.192413664 +0000 UTC m=+2594.171996022" Nov 01 01:06:44.205665 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:06:44.205578 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d157165_8ae9_465a_ad2c_6c647bd01025.slice/crio-1f93b149941d735c5c2a557a1ddb06fffb1c1320505ef2b0ea8f908dedad553a WatchSource:0}: Error finding container 1f93b149941d735c5c2a557a1ddb06fffb1c1320505ef2b0ea8f908dedad553a: Status 404 returned error can't find the container with id 1f93b149941d735c5c2a557a1ddb06fffb1c1320505ef2b0ea8f908dedad553a Nov 01 01:06:44.233219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.233194 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92"] Nov 01 01:06:44.282646 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.282583 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 01:06:44.282809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:44.282687 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 01:06:45.088513 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:45.088474 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" event={"ID":"6d157165-8ae9-465a-ad2c-6c647bd01025","Type":"ContainerStarted","Data":"6eb85a902ed06f3e5aee740ad870538351e137a70da397e448a44e62e35c0a34"} Nov 01 01:06:45.089001 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:45.088567 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" event={"ID":"6d157165-8ae9-465a-ad2c-6c647bd01025","Type":"ContainerStarted","Data":"1f93b149941d735c5c2a557a1ddb06fffb1c1320505ef2b0ea8f908dedad553a"} Nov 01 01:06:45.090056 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:45.090033 4910 generic.go:356] "Generic (PLEG): container finished" podID="9235253a-2fac-4fa5-89ef-f478c11c5fca" containerID="fa28c15a18b46c2c721fea2aeb6cf8fb4c7b32e57869c772165ff445ee69a769" exitCode=0 Nov 01 01:06:45.090176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:45.090103 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" event={"ID":"9235253a-2fac-4fa5-89ef-f478c11c5fca","Type":"ContainerDied","Data":"fa28c15a18b46c2c721fea2aeb6cf8fb4c7b32e57869c772165ff445ee69a769"} Nov 01 01:06:45.207307 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:45.207256 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" podStartSLOduration=2.207243079 podStartE2EDuration="2.207243079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:45.204464931 +0000 UTC m=+2595.184047266" watchObservedRunningTime="2026-11-01 01:06:45.207243079 +0000 UTC m=+2595.186825414" Nov 01 01:06:45.283039 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:45.283008 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 01:06:45.283204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:45.283056 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 01:06:46.099705 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.099672 4910 generic.go:356] "Generic (PLEG): container finished" podID="6d157165-8ae9-465a-ad2c-6c647bd01025" containerID="6eb85a902ed06f3e5aee740ad870538351e137a70da397e448a44e62e35c0a34" exitCode=0 Nov 01 01:06:46.100154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.099773 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" event={"ID":"6d157165-8ae9-465a-ad2c-6c647bd01025","Type":"ContainerDied","Data":"6eb85a902ed06f3e5aee740ad870538351e137a70da397e448a44e62e35c0a34"} Nov 01 01:06:46.271734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.271709 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:46.282868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.282838 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 01 01:06:46.282986 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.282891 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 01 01:06:46.416177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.416089 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9235253a-2fac-4fa5-89ef-f478c11c5fca-kube-api-access-nrhcw\" (UniqueName: \"kubernetes.io/projected/9235253a-2fac-4fa5-89ef-f478c11c5fca-kube-api-access-nrhcw\") pod \"9235253a-2fac-4fa5-89ef-f478c11c5fca\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " Nov 01 01:06:46.416177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.416129 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-xks-ca\") pod \"9235253a-2fac-4fa5-89ef-f478c11c5fca\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " Nov 01 01:06:46.416416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.416209 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-openshift-ca\") pod \"9235253a-2fac-4fa5-89ef-f478c11c5fca\" (UID: \"9235253a-2fac-4fa5-89ef-f478c11c5fca\") " Nov 01 01:06:46.416995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.416968 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-openshift-ca" pod "9235253a-2fac-4fa5-89ef-f478c11c5fca" (UID: "9235253a-2fac-4fa5-89ef-f478c11c5fca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:46.419065 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.419040 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-xks-ca" pod "9235253a-2fac-4fa5-89ef-f478c11c5fca" (UID: "9235253a-2fac-4fa5-89ef-f478c11c5fca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:06:46.419431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.419404 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9235253a-2fac-4fa5-89ef-f478c11c5fca-kube-api-access-nrhcw" pod "9235253a-2fac-4fa5-89ef-f478c11c5fca" (UID: "9235253a-2fac-4fa5-89ef-f478c11c5fca"). InnerVolumeSpecName "kube-api-access-nrhcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:46.518303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.518264 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrhcw\" (UniqueName: \"kubernetes.io/projected/9235253a-2fac-4fa5-89ef-f478c11c5fca-kube-api-access-nrhcw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:46.518303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.518296 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:46.518303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:46.518310 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235253a-2fac-4fa5-89ef-f478c11c5fca-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.110308 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.110277 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" Nov 01 01:06:47.110772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.110313 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr" event={"ID":"9235253a-2fac-4fa5-89ef-f478c11c5fca","Type":"ContainerDied","Data":"f3d695d34292feffce54656c1833e76760ae98516034ed6885fd50225f2f4c80"} Nov 01 01:06:47.110772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.110370 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3d695d34292feffce54656c1833e76760ae98516034ed6885fd50225f2f4c80" Nov 01 01:06:47.294862 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.293318 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:47.295179 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.295153 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:47.296202 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.296179 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:06:47.344117 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.344090 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:47.346215 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.346195 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 01:06:47.429306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429276 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d157165-8ae9-465a-ad2c-6c647bd01025-kube-api-access-zc77c\" (UniqueName: \"kubernetes.io/projected/6d157165-8ae9-465a-ad2c-6c647bd01025-kube-api-access-zc77c\") pod \"6d157165-8ae9-465a-ad2c-6c647bd01025\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " Nov 01 01:06:47.429458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429323 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-kube-api-access-v8crc\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-kube-api-access-v8crc\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429363 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-data\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429393 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-podinfo\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429458 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429441 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-envoy\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429471 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-socket\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429517 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cf8936-fd4f-4915-acf0-c4cdabc12636-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf8936-fd4f-4915-acf0-c4cdabc12636-istiod-ca-cert\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429537 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-credential-socket\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429556 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-token\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-token\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429575 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-certs\") pod \"51cf8936-fd4f-4915-acf0-c4cdabc12636\" (UID: \"51cf8936-fd4f-4915-acf0-c4cdabc12636\") " Nov 01 01:06:47.429734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429616 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-openshift-ca\") pod \"6d157165-8ae9-465a-ad2c-6c647bd01025\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " Nov 01 01:06:47.430064 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.429831 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-xks-ca\") pod \"6d157165-8ae9-465a-ad2c-6c647bd01025\" (UID: \"6d157165-8ae9-465a-ad2c-6c647bd01025\") " Nov 01 01:06:47.430398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.430314 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-data" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:47.431702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.430682 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cf8936-fd4f-4915-acf0-c4cdabc12636-istiod-ca-cert" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:47.431807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.430752 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-certs" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:47.431807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.430801 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-socket" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:47.431807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.430915 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-credential-socket" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:47.431807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.431459 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-openshift-ca" pod "6d157165-8ae9-465a-ad2c-6c647bd01025" (UID: "6d157165-8ae9-465a-ad2c-6c647bd01025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:47.431807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.430850 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.432965 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.432862 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d157165-8ae9-465a-ad2c-6c647bd01025-kube-api-access-zc77c" pod "6d157165-8ae9-465a-ad2c-6c647bd01025" (UID: "6d157165-8ae9-465a-ad2c-6c647bd01025"). InnerVolumeSpecName "kube-api-access-zc77c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:47.433129 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.433104 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-podinfo" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:06:47.433349 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.433299 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-kube-api-access-v8crc" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "kube-api-access-v8crc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:47.433514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.433491 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-token" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:47.433756 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.433713 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-envoy" pod "51cf8936-fd4f-4915-acf0-c4cdabc12636" (UID: "51cf8936-fd4f-4915-acf0-c4cdabc12636"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:06:47.433948 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.433928 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-xks-ca" pod "6d157165-8ae9-465a-ad2c-6c647bd01025" (UID: "6d157165-8ae9-465a-ad2c-6c647bd01025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:06:47.532810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532773 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.532810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532811 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc77c\" (UniqueName: \"kubernetes.io/projected/6d157165-8ae9-465a-ad2c-6c647bd01025-kube-api-access-zc77c\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532828 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8crc\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-kube-api-access-v8crc\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532842 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532856 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532869 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532883 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf8936-fd4f-4915-acf0-c4cdabc12636-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532897 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532908 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf8936-fd4f-4915-acf0-c4cdabc12636-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532922 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf8936-fd4f-4915-acf0-c4cdabc12636-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:47.533076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:47.532937 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d157165-8ae9-465a-ad2c-6c647bd01025-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:48.120595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.120566 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" Nov 01 01:06:48.120595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.120589 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92" event={"ID":"6d157165-8ae9-465a-ad2c-6c647bd01025","Type":"ContainerDied","Data":"1f93b149941d735c5c2a557a1ddb06fffb1c1320505ef2b0ea8f908dedad553a"} Nov 01 01:06:48.121195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.120679 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f93b149941d735c5c2a557a1ddb06fffb1c1320505ef2b0ea8f908dedad553a" Nov 01 01:06:48.122639 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.122599 4910 generic.go:356] "Generic (PLEG): container finished" podID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerID="b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93" exitCode=0 Nov 01 01:06:48.122760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.122655 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" Nov 01 01:06:48.122760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.122683 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" event={"ID":"51cf8936-fd4f-4915-acf0-c4cdabc12636","Type":"ContainerDied","Data":"b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93"} Nov 01 01:06:48.122760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.122740 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l" event={"ID":"51cf8936-fd4f-4915-acf0-c4cdabc12636","Type":"ContainerDied","Data":"45fa72ba6a33e18ef18e8f2c178b65dbd6c1a39a4644ec55e8db7a9cd6a21398"} Nov 01 01:06:48.122913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.122775 4910 scope.go:122] "RemoveContainer" containerID="b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93" Nov 01 01:06:48.135018 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.135002 4910 scope.go:122] "RemoveContainer" containerID="4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018" Nov 01 01:06:48.144919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.144900 4910 scope.go:122] "RemoveContainer" containerID="b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93" Nov 01 01:06:48.145247 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:06:48.145222 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93\": container with ID starting with b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93 not found: ID does not exist" containerID="b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93" Nov 01 01:06:48.145327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.145256 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93"} err="failed to get container status \"b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93\": rpc error: code = NotFound desc = could not find container \"b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93\": container with ID starting with b006473c0654067920540add4bfb6df145fa4a6ea5d432f63a9ae8cf41e0cc93 not found: ID does not exist" Nov 01 01:06:48.145327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.145274 4910 scope.go:122] "RemoveContainer" containerID="4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018" Nov 01 01:06:48.145514 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:06:48.145497 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018\": container with ID starting with 4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018 not found: ID does not exist" containerID="4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018" Nov 01 01:06:48.145559 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.145521 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018"} err="failed to get container status \"4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018\": rpc error: code = NotFound desc = could not find container \"4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018\": container with ID starting with 4af6eb739c1d4f01d20b5dd52305cf0d5df4c894c4a9195e1fa7d5a02f6a8018 not found: ID does not exist" Nov 01 01:06:48.290843 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.290753 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l"] Nov 01 01:06:48.295135 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:48.295107 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f97e89-openshift-default-58b559fdd5-9fb8l"] Nov 01 01:06:50.194450 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:50.194416 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" path="/var/lib/kubelet/pods/51cf8936-fd4f-4915-acf0-c4cdabc12636/volumes" Nov 01 01:06:52.135203 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.135174 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 01:06:52.158187 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.158159 4910 generic.go:356] "Generic (PLEG): container finished" podID="56a036e7-5a71-4ff4-9a77-bc23f2249437" containerID="cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603" exitCode=0 Nov 01 01:06:52.158374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.158218 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" Nov 01 01:06:52.158374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.158245 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" event={"ID":"56a036e7-5a71-4ff4-9a77-bc23f2249437","Type":"ContainerDied","Data":"cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603"} Nov 01 01:06:52.158374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.158360 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq" event={"ID":"56a036e7-5a71-4ff4-9a77-bc23f2249437","Type":"ContainerDied","Data":"a05aaccaa68cfe4f0b6e7aff3fc17a2ecc3b8a835e01a6c775312d6eb26ed05f"} Nov 01 01:06:52.158564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.158394 4910 scope.go:122] "RemoveContainer" containerID="cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603" Nov 01 01:06:52.168695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.168675 4910 scope.go:122] "RemoveContainer" containerID="cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603" Nov 01 01:06:52.169019 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:06:52.168996 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603\": container with ID starting with cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603 not found: ID does not exist" containerID="cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603" Nov 01 01:06:52.169119 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.169028 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603"} err="failed to get container status \"cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603\": rpc error: code = NotFound desc = could not find container \"cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603\": container with ID starting with cdab4eff356fef22301a9f798515afd236c1a8301f936ae441ed4e59a1630603 not found: ID does not exist" Nov 01 01:06:52.303691 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.303610 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a036e7-5a71-4ff4-9a77-bc23f2249437-kube-api-access-q2csm\" (UniqueName: \"kubernetes.io/projected/56a036e7-5a71-4ff4-9a77-bc23f2249437-kube-api-access-q2csm\") pod \"56a036e7-5a71-4ff4-9a77-bc23f2249437\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " Nov 01 01:06:52.303879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.303752 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a036e7-5a71-4ff4-9a77-bc23f2249437-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a036e7-5a71-4ff4-9a77-bc23f2249437-plugins-config-volume\") pod \"56a036e7-5a71-4ff4-9a77-bc23f2249437\" (UID: \"56a036e7-5a71-4ff4-9a77-bc23f2249437\") " Nov 01 01:06:52.304272 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.304240 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a036e7-5a71-4ff4-9a77-bc23f2249437-plugins-config-volume" pod "56a036e7-5a71-4ff4-9a77-bc23f2249437" (UID: "56a036e7-5a71-4ff4-9a77-bc23f2249437"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:06:52.304561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.304481 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a036e7-5a71-4ff4-9a77-bc23f2249437-plugins-config-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:52.306437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.306409 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a036e7-5a71-4ff4-9a77-bc23f2249437-kube-api-access-q2csm" pod "56a036e7-5a71-4ff4-9a77-bc23f2249437" (UID: "56a036e7-5a71-4ff4-9a77-bc23f2249437"). InnerVolumeSpecName "kube-api-access-q2csm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:06:52.406130 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.406093 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2csm\" (UniqueName: \"kubernetes.io/projected/56a036e7-5a71-4ff4-9a77-bc23f2249437-kube-api-access-q2csm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:06:52.582220 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.582132 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq"] Nov 01 01:06:52.588617 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:52.588590 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4143b7-749cd5f7f6-v9pxq"] Nov 01 01:06:53.763019 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.762983 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9"] Nov 01 01:06:53.763913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763895 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9235253a-2fac-4fa5-89ef-f478c11c5fca" containerName="revoke-keys" Nov 01 01:06:53.763995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763917 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9235253a-2fac-4fa5-89ef-f478c11c5fca" containerName="revoke-keys" Nov 01 01:06:53.763995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763941 4910 cpu_manager.go:414] "containerMap: removing container" podUID="56a036e7-5a71-4ff4-9a77-bc23f2249437" containerName="payload-processing" Nov 01 01:06:53.763995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763949 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a036e7-5a71-4ff4-9a77-bc23f2249437" containerName="payload-processing" Nov 01 01:06:53.763995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763959 4910 cpu_manager.go:414] "containerMap: removing container" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" Nov 01 01:06:53.763995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763966 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" Nov 01 01:06:53.763995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763976 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6d157165-8ae9-465a-ad2c-6c647bd01025" containerName="revoke-keys" Nov 01 01:06:53.763995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.763983 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d157165-8ae9-465a-ad2c-6c647bd01025" containerName="revoke-keys" Nov 01 01:06:53.764252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.764024 4910 cpu_manager.go:414] "containerMap: removing container" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" Nov 01 01:06:53.764252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.764031 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" Nov 01 01:06:53.764252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.764189 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9235253a-2fac-4fa5-89ef-f478c11c5fca" containerName="revoke-keys" Nov 01 01:06:53.764252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.764201 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a036e7-5a71-4ff4-9a77-bc23f2249437" containerName="payload-processing" Nov 01 01:06:53.764252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.764211 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" Nov 01 01:06:53.764252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.764235 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d157165-8ae9-465a-ad2c-6c647bd01025" containerName="revoke-keys" Nov 01 01:06:53.764252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.764251 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cf8936-fd4f-4915-acf0-c4cdabc12636" containerName="istio-proxy" Nov 01 01:06:53.769587 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.769569 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.878539 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.878505 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9"] Nov 01 01:06:53.934034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.933997 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0452ae-29cc-4144-985e-deb83634b42b-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934118 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934184 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934296 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934208 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934296 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934240 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0452ae-29cc-4144-985e-deb83634b42b-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934322 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934444 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934525 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934487 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz68g\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-kube-api-access-hz68g\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:53.934581 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:53.934533 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035274 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035191 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035274 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035231 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0452ae-29cc-4144-985e-deb83634b42b-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035275 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035346 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035365 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz68g\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-kube-api-access-hz68g\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035386 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035412 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0452ae-29cc-4144-985e-deb83634b42b-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035481 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035540 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035797 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-data\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.035998 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.035948 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-credential-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.036067 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.036037 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-socket\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.036181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.036160 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0452ae-29cc-4144-985e-deb83634b42b-istiod-ca-cert\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.036401 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.036364 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-certs\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.038480 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.038451 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-envoy\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.038617 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.038608 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0452ae-29cc-4144-985e-deb83634b42b-istio-podinfo\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.096868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.096840 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-istio-token\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.097013 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.096988 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz68g\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-kube-api-access-hz68g\") pod \"e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.192392 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.192361 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a036e7-5a71-4ff4-9a77-bc23f2249437" path="/var/lib/kubelet/pods/56a036e7-5a71-4ff4-9a77-bc23f2249437/volumes" Nov 01 01:06:54.385003 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.384917 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:54.842438 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:06:54.842407 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0452ae_29cc_4144_985e_deb83634b42b.slice/crio-2a72abc644ef1e69f1b3c5ecc7feabad34b6c58fa7a069eaf59be5f58de2e7d8 WatchSource:0}: Error finding container 2a72abc644ef1e69f1b3c5ecc7feabad34b6c58fa7a069eaf59be5f58de2e7d8: Status 404 returned error can't find the container with id 2a72abc644ef1e69f1b3c5ecc7feabad34b6c58fa7a069eaf59be5f58de2e7d8 Nov 01 01:06:54.843061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.842993 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9"] Nov 01 01:06:54.845136 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.845092 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:54.845222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.845165 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:54.845222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:54.845198 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:06:55.185870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:55.185776 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" event={"ID":"5d0452ae-29cc-4144-985e-deb83634b42b","Type":"ContainerStarted","Data":"83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead"} Nov 01 01:06:55.185870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:55.185825 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" event={"ID":"5d0452ae-29cc-4144-985e-deb83634b42b","Type":"ContainerStarted","Data":"2a72abc644ef1e69f1b3c5ecc7feabad34b6c58fa7a069eaf59be5f58de2e7d8"} Nov 01 01:06:55.310275 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:55.310211 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podStartSLOduration=2.310191531 podStartE2EDuration="2.310191531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:06:55.307544344 +0000 UTC m=+2605.287126680" watchObservedRunningTime="2026-11-01 01:06:55.310191531 +0000 UTC m=+2605.289773867" Nov 01 01:06:55.386848 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:55.386806 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:06:55.388851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:55.388825 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 01:06:55.388980 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:55.388873 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 01:06:56.396663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:56.394587 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 01:06:56.396663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:56.394683 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 01:06:57.402701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:57.401606 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 01:06:57.402701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:57.401686 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 01:06:58.388078 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:58.388015 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 01:06:58.388288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:58.388087 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 01:06:59.385469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:59.385427 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 01:06:59.386210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:59.385480 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 01:06:59.982974 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:59.982927 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq"] Nov 01 01:06:59.987401 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:59.987372 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:06:59.994525 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:59.994499 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:06:59.994711 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:59.994686 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:06:59.994761 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:06:59.994731 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kqx5\" (UniqueName: \"kubernetes.io/projected/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-kube-api-access-2kqx5\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.017618 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.017588 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf"] Nov 01 01:07:00.021745 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.021725 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.093024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.087150 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:07:00.093024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.087804 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:07:00.100647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.097309 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.100647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.097423 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.100647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.097477 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.100647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.097507 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqx5\" (UniqueName: \"kubernetes.io/projected/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-kube-api-access-2kqx5\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.100647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.097568 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/7a8291ba-087b-486d-a4b8-c4a3708f5a41-kube-api-access-zlvmr\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.100647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.097599 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.101071 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.100677 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.102102 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.102075 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq"] Nov 01 01:07:00.104925 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.104890 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.109072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.109048 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf"] Nov 01 01:07:00.188566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.188512 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqx5\" (UniqueName: \"kubernetes.io/projected/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-kube-api-access-2kqx5\") pod \"maas-api-revoke-sub-d4808cb9-jjrjq\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.199772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.199325 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.199772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.199397 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/7a8291ba-087b-486d-a4b8-c4a3708f5a41-kube-api-access-zlvmr\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.199772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.199428 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.201935 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.200789 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.202475 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.202447 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.272652 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.269568 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/7a8291ba-087b-486d-a4b8-c4a3708f5a41-kube-api-access-zlvmr\") pod \"maas-api-revoke-sub-8a8a1d22-2v7zf\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.304616 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.304584 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:00.385679 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.385533 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 01:07:00.385679 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.385594 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 01:07:00.410292 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.410244 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:00.783779 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.783244 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq"] Nov 01 01:07:00.888398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:00.888367 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf"] Nov 01 01:07:01.504350 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:01.504307 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 01 01:07:01.505432 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:01.505379 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 01 01:07:01.521010 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:01.520899 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" event={"ID":"4ddcea4c-009d-4f8d-ae47-e06d0e265be9","Type":"ContainerStarted","Data":"0b0d70a7e86ec74374c6b15e7ac3a786db421c8c8dae2285d1740ed800cceb91"} Nov 01 01:07:01.523039 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:01.523011 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" event={"ID":"7a8291ba-087b-486d-a4b8-c4a3708f5a41","Type":"ContainerStarted","Data":"aa68b50ea894a8e898f255352268a75805c1b4c809e3a1e58f94bfdef8d1973a"} Nov 01 01:07:02.505706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:02.494280 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:07:02.505706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:02.495850 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:07:02.505706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:02.501682 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:07:02.533114 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:02.533079 4910 generic.go:356] "Generic (PLEG): container finished" podID="4ddcea4c-009d-4f8d-ae47-e06d0e265be9" containerID="1294aea4e75ca22b74134ba2bb7799db4af0ef8338aa4b8a4dcc32fc7754601e" exitCode=0 Nov 01 01:07:02.533328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:02.533244 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" event={"ID":"4ddcea4c-009d-4f8d-ae47-e06d0e265be9","Type":"ContainerDied","Data":"1294aea4e75ca22b74134ba2bb7799db4af0ef8338aa4b8a4dcc32fc7754601e"} Nov 01 01:07:02.535224 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:02.535202 4910 generic.go:356] "Generic (PLEG): container finished" podID="7a8291ba-087b-486d-a4b8-c4a3708f5a41" containerID="df249ab79aca003715b4ef9bdbf196e5f01e1ad9e76a9b2720357d9d3e36486e" exitCode=0 Nov 01 01:07:02.535771 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:02.535518 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" event={"ID":"7a8291ba-087b-486d-a4b8-c4a3708f5a41","Type":"ContainerDied","Data":"df249ab79aca003715b4ef9bdbf196e5f01e1ad9e76a9b2720357d9d3e36486e"} Nov 01 01:07:03.757078 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.757054 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:03.761653 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.761631 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:03.825345 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.825271 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-xks-ca\") pod \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " Nov 01 01:07:03.825345 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.825342 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8291ba-087b-486d-a4b8-c4a3708f5a41-kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/7a8291ba-087b-486d-a4b8-c4a3708f5a41-kube-api-access-zlvmr\") pod \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " Nov 01 01:07:03.825591 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.825402 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-xks-ca\") pod \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " Nov 01 01:07:03.825591 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.825449 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-openshift-ca\") pod \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\" (UID: \"7a8291ba-087b-486d-a4b8-c4a3708f5a41\") " Nov 01 01:07:03.825591 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.825510 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-kube-api-access-2kqx5\" (UniqueName: \"kubernetes.io/projected/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-kube-api-access-2kqx5\") pod \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " Nov 01 01:07:03.825591 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.825539 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-openshift-ca\") pod \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\" (UID: \"4ddcea4c-009d-4f8d-ae47-e06d0e265be9\") " Nov 01 01:07:03.825950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.825925 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-openshift-ca" pod "7a8291ba-087b-486d-a4b8-c4a3708f5a41" (UID: "7a8291ba-087b-486d-a4b8-c4a3708f5a41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:03.826132 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.826106 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:03.826401 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.826195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-openshift-ca" pod "4ddcea4c-009d-4f8d-ae47-e06d0e265be9" (UID: "4ddcea4c-009d-4f8d-ae47-e06d0e265be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:03.827903 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.827879 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-xks-ca" pod "4ddcea4c-009d-4f8d-ae47-e06d0e265be9" (UID: "4ddcea4c-009d-4f8d-ae47-e06d0e265be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:03.828038 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.827950 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-kube-api-access-2kqx5" pod "4ddcea4c-009d-4f8d-ae47-e06d0e265be9" (UID: "4ddcea4c-009d-4f8d-ae47-e06d0e265be9"). InnerVolumeSpecName "kube-api-access-2kqx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:03.828300 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.828280 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-xks-ca" pod "7a8291ba-087b-486d-a4b8-c4a3708f5a41" (UID: "7a8291ba-087b-486d-a4b8-c4a3708f5a41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:03.828362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.828324 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8291ba-087b-486d-a4b8-c4a3708f5a41-kube-api-access-zlvmr" pod "7a8291ba-087b-486d-a4b8-c4a3708f5a41" (UID: "7a8291ba-087b-486d-a4b8-c4a3708f5a41"). InnerVolumeSpecName "kube-api-access-zlvmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:03.927512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.927480 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:03.927512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.927508 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kqx5\" (UniqueName: \"kubernetes.io/projected/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-kube-api-access-2kqx5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:03.927512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.927519 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ddcea4c-009d-4f8d-ae47-e06d0e265be9-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:03.927778 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.927528 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8291ba-087b-486d-a4b8-c4a3708f5a41-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:03.927778 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:03.927537 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/7a8291ba-087b-486d-a4b8-c4a3708f5a41-kube-api-access-zlvmr\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:04.552101 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:04.552065 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" event={"ID":"4ddcea4c-009d-4f8d-ae47-e06d0e265be9","Type":"ContainerDied","Data":"0b0d70a7e86ec74374c6b15e7ac3a786db421c8c8dae2285d1740ed800cceb91"} Nov 01 01:07:04.552287 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:04.552131 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b0d70a7e86ec74374c6b15e7ac3a786db421c8c8dae2285d1740ed800cceb91" Nov 01 01:07:04.552287 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:04.552079 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq" Nov 01 01:07:04.554342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:04.554317 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" Nov 01 01:07:04.554342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:04.554325 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf" event={"ID":"7a8291ba-087b-486d-a4b8-c4a3708f5a41","Type":"ContainerDied","Data":"aa68b50ea894a8e898f255352268a75805c1b4c809e3a1e58f94bfdef8d1973a"} Nov 01 01:07:04.554511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:04.554379 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa68b50ea894a8e898f255352268a75805c1b4c809e3a1e58f94bfdef8d1973a" Nov 01 01:07:07.797681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:07.797640 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv"] Nov 01 01:07:07.847243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:07.847213 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-44c9bf-74blv"] Nov 01 01:07:08.194899 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:08.194807 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36b4faf-9322-41c6-bf9c-8f6af66f9c04" path="/var/lib/kubelet/pods/f36b4faf-9322-41c6-bf9c-8f6af66f9c04/volumes" Nov 01 01:07:13.103913 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:07:13.103865 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:07:15.661913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.661877 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q"] Nov 01 01:07:15.662505 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.662490 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7a8291ba-087b-486d-a4b8-c4a3708f5a41" containerName="revoke-keys" Nov 01 01:07:15.662558 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.662507 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8291ba-087b-486d-a4b8-c4a3708f5a41" containerName="revoke-keys" Nov 01 01:07:15.662558 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.662530 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4ddcea4c-009d-4f8d-ae47-e06d0e265be9" containerName="revoke-keys" Nov 01 01:07:15.662558 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.662536 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddcea4c-009d-4f8d-ae47-e06d0e265be9" containerName="revoke-keys" Nov 01 01:07:15.662695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.662643 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddcea4c-009d-4f8d-ae47-e06d0e265be9" containerName="revoke-keys" Nov 01 01:07:15.662695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.662668 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8291ba-087b-486d-a4b8-c4a3708f5a41" containerName="revoke-keys" Nov 01 01:07:15.667985 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.667953 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.714042 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.714005 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7pp\" (UniqueName: \"kubernetes.io/projected/9fcaa2cb-297e-485d-b288-0707b1111692-kube-api-access-jq7pp\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.714190 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.714073 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.714241 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.714190 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.724358 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.724328 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q"] Nov 01 01:07:15.768370 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.768333 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:07:15.768566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.768471 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:07:15.816160 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.816123 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7pp\" (UniqueName: \"kubernetes.io/projected/9fcaa2cb-297e-485d-b288-0707b1111692-kube-api-access-jq7pp\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.816369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.816173 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.816369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.816209 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.817037 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.817013 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.818949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.818925 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.877138 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.877099 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7pp\" (UniqueName: \"kubernetes.io/projected/9fcaa2cb-297e-485d-b288-0707b1111692-kube-api-access-jq7pp\") pod \"maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:15.986007 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:15.985920 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:16.367918 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:16.367885 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q"] Nov 01 01:07:16.368766 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:07:16.368736 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcaa2cb_297e_485d_b288_0707b1111692.slice/crio-3dd705f745296bc707aca5c1a5157145dbac353cca97cc92d800f21d5e1a3d4d WatchSource:0}: Error finding container 3dd705f745296bc707aca5c1a5157145dbac353cca97cc92d800f21d5e1a3d4d: Status 404 returned error can't find the container with id 3dd705f745296bc707aca5c1a5157145dbac353cca97cc92d800f21d5e1a3d4d Nov 01 01:07:16.680386 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:16.680357 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" event={"ID":"9fcaa2cb-297e-485d-b288-0707b1111692","Type":"ContainerStarted","Data":"40f236470a8e64cbdd7a2c9bb984d996f8ec57d451eb613c4ae87291435b469c"} Nov 01 01:07:16.680822 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:16.680405 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" event={"ID":"9fcaa2cb-297e-485d-b288-0707b1111692","Type":"ContainerStarted","Data":"3dd705f745296bc707aca5c1a5157145dbac353cca97cc92d800f21d5e1a3d4d"} Nov 01 01:07:16.803122 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:16.803070 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" podStartSLOduration=1.803055417 podStartE2EDuration="1.803055417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:16.799867967 +0000 UTC m=+2626.779450301" watchObservedRunningTime="2026-11-01 01:07:16.803055417 +0000 UTC m=+2626.782637760" Nov 01 01:07:17.692990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:17.692900 4910 generic.go:356] "Generic (PLEG): container finished" podID="9fcaa2cb-297e-485d-b288-0707b1111692" containerID="40f236470a8e64cbdd7a2c9bb984d996f8ec57d451eb613c4ae87291435b469c" exitCode=0 Nov 01 01:07:17.693462 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:17.692989 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" event={"ID":"9fcaa2cb-297e-485d-b288-0707b1111692","Type":"ContainerDied","Data":"40f236470a8e64cbdd7a2c9bb984d996f8ec57d451eb613c4ae87291435b469c"} Nov 01 01:07:18.866955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:18.866926 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:18.955412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:18.955334 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-xks-ca\") pod \"9fcaa2cb-297e-485d-b288-0707b1111692\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " Nov 01 01:07:18.955587 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:18.955495 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcaa2cb-297e-485d-b288-0707b1111692-kube-api-access-jq7pp\" (UniqueName: \"kubernetes.io/projected/9fcaa2cb-297e-485d-b288-0707b1111692-kube-api-access-jq7pp\") pod \"9fcaa2cb-297e-485d-b288-0707b1111692\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " Nov 01 01:07:18.955587 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:18.955537 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-openshift-ca\") pod \"9fcaa2cb-297e-485d-b288-0707b1111692\" (UID: \"9fcaa2cb-297e-485d-b288-0707b1111692\") " Nov 01 01:07:18.955982 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:18.955958 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-openshift-ca" pod "9fcaa2cb-297e-485d-b288-0707b1111692" (UID: "9fcaa2cb-297e-485d-b288-0707b1111692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:18.958143 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:18.958115 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-xks-ca" pod "9fcaa2cb-297e-485d-b288-0707b1111692" (UID: "9fcaa2cb-297e-485d-b288-0707b1111692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:18.958463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:18.958435 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcaa2cb-297e-485d-b288-0707b1111692-kube-api-access-jq7pp" pod "9fcaa2cb-297e-485d-b288-0707b1111692" (UID: "9fcaa2cb-297e-485d-b288-0707b1111692"). InnerVolumeSpecName "kube-api-access-jq7pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:19.057358 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:19.057329 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7pp\" (UniqueName: \"kubernetes.io/projected/9fcaa2cb-297e-485d-b288-0707b1111692-kube-api-access-jq7pp\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:19.057358 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:19.057358 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:19.057358 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:19.057374 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcaa2cb-297e-485d-b288-0707b1111692-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:19.710484 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:19.710457 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" Nov 01 01:07:19.710671 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:19.710494 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q" event={"ID":"9fcaa2cb-297e-485d-b288-0707b1111692","Type":"ContainerDied","Data":"3dd705f745296bc707aca5c1a5157145dbac353cca97cc92d800f21d5e1a3d4d"} Nov 01 01:07:19.710671 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:19.710543 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dd705f745296bc707aca5c1a5157145dbac353cca97cc92d800f21d5e1a3d4d" Nov 01 01:07:20.148825 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:20.148791 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl"] Nov 01 01:07:28.258163 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.258131 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8"] Nov 01 01:07:28.258748 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.258732 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9fcaa2cb-297e-485d-b288-0707b1111692" containerName="revoke-keys" Nov 01 01:07:28.258748 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.258749 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcaa2cb-297e-485d-b288-0707b1111692" containerName="revoke-keys" Nov 01 01:07:28.258889 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.258880 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcaa2cb-297e-485d-b288-0707b1111692" containerName="revoke-keys" Nov 01 01:07:28.262755 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.262735 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.303679 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.303616 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.303826 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.303785 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.304046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.304012 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvf9\" (UniqueName: \"kubernetes.io/projected/1982ee14-47f4-4a02-9040-6b1200ca9864-kube-api-access-xfvf9\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.318655 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.316747 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:07:28.318655 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.318515 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn"] Nov 01 01:07:28.322940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.322921 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.363803 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.363773 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:07:28.377831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.377793 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8"] Nov 01 01:07:28.408069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.406137 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee42edb-342d-478a-a090-368586987ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.408069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.406210 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee42edb-342d-478a-a090-368586987ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.408069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.406254 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chfz2\" (UniqueName: \"kubernetes.io/projected/5ee42edb-342d-478a-a090-368586987ac9-kube-api-access-chfz2\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.408069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.406305 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvf9\" (UniqueName: \"kubernetes.io/projected/1982ee14-47f4-4a02-9040-6b1200ca9864-kube-api-access-xfvf9\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.408069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.406344 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.408069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.406402 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.408069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.407267 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.437347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.437255 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.442912 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.442886 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn"] Nov 01 01:07:28.470612 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.470585 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvf9\" (UniqueName: \"kubernetes.io/projected/1982ee14-47f4-4a02-9040-6b1200ca9864-kube-api-access-xfvf9\") pod \"maas-api-revoke-sub-2484fb0e-s6ts8\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.507284 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.507256 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee42edb-342d-478a-a090-368586987ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.507431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.507313 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee42edb-342d-478a-a090-368586987ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.507431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.507345 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chfz2\" (UniqueName: \"kubernetes.io/projected/5ee42edb-342d-478a-a090-368586987ac9-kube-api-access-chfz2\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.508064 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.508040 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee42edb-342d-478a-a090-368586987ac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.509800 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.509743 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee42edb-342d-478a-a090-368586987ac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.569032 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.569000 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chfz2\" (UniqueName: \"kubernetes.io/projected/5ee42edb-342d-478a-a090-368586987ac9-kube-api-access-chfz2\") pod \"maas-api-revoke-sub-9b36c398-4hbqn\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:28.577743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.577713 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:28.639162 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:28.639128 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:29.040401 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:29.040330 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn"] Nov 01 01:07:29.078837 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:07:29.078802 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee42edb_342d_478a_a090_368586987ac9.slice/crio-5ec7f374814a9190fc0946fc7ea3c95a529953ccaec3e829508286c2fdba8cb0 WatchSource:0}: Error finding container 5ec7f374814a9190fc0946fc7ea3c95a529953ccaec3e829508286c2fdba8cb0: Status 404 returned error can't find the container with id 5ec7f374814a9190fc0946fc7ea3c95a529953ccaec3e829508286c2fdba8cb0 Nov 01 01:07:29.090800 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:29.090776 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8"] Nov 01 01:07:29.093836 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:07:29.093805 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1982ee14_47f4_4a02_9040_6b1200ca9864.slice/crio-9f4ad6d4f1b92f6aea15e443ce1dc513a2bbb0df8bd9fb0a8ae5c203a6bf8a9f WatchSource:0}: Error finding container 9f4ad6d4f1b92f6aea15e443ce1dc513a2bbb0df8bd9fb0a8ae5c203a6bf8a9f: Status 404 returned error can't find the container with id 9f4ad6d4f1b92f6aea15e443ce1dc513a2bbb0df8bd9fb0a8ae5c203a6bf8a9f Nov 01 01:07:29.883088 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:29.883049 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" event={"ID":"1982ee14-47f4-4a02-9040-6b1200ca9864","Type":"ContainerStarted","Data":"886ada4b8f2424d6f05a264e2846639e8133e86ecd90a704d4f93cc69526bd1e"} Nov 01 01:07:29.883547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:29.883110 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" event={"ID":"1982ee14-47f4-4a02-9040-6b1200ca9864","Type":"ContainerStarted","Data":"9f4ad6d4f1b92f6aea15e443ce1dc513a2bbb0df8bd9fb0a8ae5c203a6bf8a9f"} Nov 01 01:07:29.885026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:29.885001 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" event={"ID":"5ee42edb-342d-478a-a090-368586987ac9","Type":"ContainerStarted","Data":"b2610b89e48e7673f2ab6939f6e378126750fa9ef3a0672b35f488613dd84f1c"} Nov 01 01:07:29.885118 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:29.885051 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" event={"ID":"5ee42edb-342d-478a-a090-368586987ac9","Type":"ContainerStarted","Data":"5ec7f374814a9190fc0946fc7ea3c95a529953ccaec3e829508286c2fdba8cb0"} Nov 01 01:07:30.004873 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.004816 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" podStartSLOduration=2.004801266 podStartE2EDuration="2.004801266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:29.999697833 +0000 UTC m=+2639.979280185" watchObservedRunningTime="2026-11-01 01:07:30.004801266 +0000 UTC m=+2639.984383654" Nov 01 01:07:30.055708 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.055677 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5"] Nov 01 01:07:30.060536 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.060510 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d5e287-gvjq5"] Nov 01 01:07:30.117316 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.117257 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" podStartSLOduration=2.117239924 podStartE2EDuration="2.117239924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:07:30.115434447 +0000 UTC m=+2640.095016808" watchObservedRunningTime="2026-11-01 01:07:30.117239924 +0000 UTC m=+2640.096822265" Nov 01 01:07:30.197176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.197133 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70068e56-d779-46c7-899e-b82506bcb1ae" path="/var/lib/kubelet/pods/70068e56-d779-46c7-899e-b82506bcb1ae/volumes" Nov 01 01:07:30.829743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.829477 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92"] Nov 01 01:07:30.833278 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.833255 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b112c158-7lh92"] Nov 01 01:07:30.909486 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.909446 4910 generic.go:356] "Generic (PLEG): container finished" podID="1982ee14-47f4-4a02-9040-6b1200ca9864" containerID="886ada4b8f2424d6f05a264e2846639e8133e86ecd90a704d4f93cc69526bd1e" exitCode=0 Nov 01 01:07:30.910051 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.909659 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" event={"ID":"1982ee14-47f4-4a02-9040-6b1200ca9864","Type":"ContainerDied","Data":"886ada4b8f2424d6f05a264e2846639e8133e86ecd90a704d4f93cc69526bd1e"} Nov 01 01:07:30.911923 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.911898 4910 generic.go:356] "Generic (PLEG): container finished" podID="5ee42edb-342d-478a-a090-368586987ac9" containerID="b2610b89e48e7673f2ab6939f6e378126750fa9ef3a0672b35f488613dd84f1c" exitCode=0 Nov 01 01:07:30.912069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:30.912009 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" event={"ID":"5ee42edb-342d-478a-a090-368586987ac9","Type":"ContainerDied","Data":"b2610b89e48e7673f2ab6939f6e378126750fa9ef3a0672b35f488613dd84f1c"} Nov 01 01:07:32.102153 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.102126 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:32.142564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.142536 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:32.199005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.198935 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d157165-8ae9-465a-ad2c-6c647bd01025" path="/var/lib/kubelet/pods/6d157165-8ae9-465a-ad2c-6c647bd01025/volumes" Nov 01 01:07:32.280697 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.280664 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1982ee14-47f4-4a02-9040-6b1200ca9864-kube-api-access-xfvf9\" (UniqueName: \"kubernetes.io/projected/1982ee14-47f4-4a02-9040-6b1200ca9864-kube-api-access-xfvf9\") pod \"1982ee14-47f4-4a02-9040-6b1200ca9864\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " Nov 01 01:07:32.280879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.280744 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee42edb-342d-478a-a090-368586987ac9-kube-api-access-chfz2\" (UniqueName: \"kubernetes.io/projected/5ee42edb-342d-478a-a090-368586987ac9-kube-api-access-chfz2\") pod \"5ee42edb-342d-478a-a090-368586987ac9\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " Nov 01 01:07:32.280879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.280769 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee42edb-342d-478a-a090-368586987ac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee42edb-342d-478a-a090-368586987ac9-maas-api-xks-ca\") pod \"5ee42edb-342d-478a-a090-368586987ac9\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " Nov 01 01:07:32.280879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.280855 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee42edb-342d-478a-a090-368586987ac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee42edb-342d-478a-a090-368586987ac9-maas-api-openshift-ca\") pod \"5ee42edb-342d-478a-a090-368586987ac9\" (UID: \"5ee42edb-342d-478a-a090-368586987ac9\") " Nov 01 01:07:32.281108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.280954 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-openshift-ca\") pod \"1982ee14-47f4-4a02-9040-6b1200ca9864\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " Nov 01 01:07:32.281108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.281033 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-xks-ca\") pod \"1982ee14-47f4-4a02-9040-6b1200ca9864\" (UID: \"1982ee14-47f4-4a02-9040-6b1200ca9864\") " Nov 01 01:07:32.281382 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.281281 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee42edb-342d-478a-a090-368586987ac9-maas-api-openshift-ca" pod "5ee42edb-342d-478a-a090-368586987ac9" (UID: "5ee42edb-342d-478a-a090-368586987ac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:32.281382 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.281357 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-openshift-ca" pod "1982ee14-47f4-4a02-9040-6b1200ca9864" (UID: "1982ee14-47f4-4a02-9040-6b1200ca9864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:32.281635 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.281593 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:32.281732 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.281617 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee42edb-342d-478a-a090-368586987ac9-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:32.283099 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.283073 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1982ee14-47f4-4a02-9040-6b1200ca9864-kube-api-access-xfvf9" pod "1982ee14-47f4-4a02-9040-6b1200ca9864" (UID: "1982ee14-47f4-4a02-9040-6b1200ca9864"). InnerVolumeSpecName "kube-api-access-xfvf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:32.283356 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.283341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-xks-ca" pod "1982ee14-47f4-4a02-9040-6b1200ca9864" (UID: "1982ee14-47f4-4a02-9040-6b1200ca9864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:32.283545 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.283524 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee42edb-342d-478a-a090-368586987ac9-maas-api-xks-ca" pod "5ee42edb-342d-478a-a090-368586987ac9" (UID: "5ee42edb-342d-478a-a090-368586987ac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:32.283678 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.283661 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee42edb-342d-478a-a090-368586987ac9-kube-api-access-chfz2" pod "5ee42edb-342d-478a-a090-368586987ac9" (UID: "5ee42edb-342d-478a-a090-368586987ac9"). InnerVolumeSpecName "kube-api-access-chfz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:32.383362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.383332 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfvf9\" (UniqueName: \"kubernetes.io/projected/1982ee14-47f4-4a02-9040-6b1200ca9864-kube-api-access-xfvf9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:32.383362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.383360 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chfz2\" (UniqueName: \"kubernetes.io/projected/5ee42edb-342d-478a-a090-368586987ac9-kube-api-access-chfz2\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:32.383557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.383372 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee42edb-342d-478a-a090-368586987ac9-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:32.383557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.383381 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1982ee14-47f4-4a02-9040-6b1200ca9864-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:32.446374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.446346 4910 scope.go:122] "RemoveContainer" containerID="5565d103378f15d1d540043a0e9726d25dc11190ca4d68d9def6bafff0fc7dfe" Nov 01 01:07:32.932267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.932230 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" event={"ID":"1982ee14-47f4-4a02-9040-6b1200ca9864","Type":"ContainerDied","Data":"9f4ad6d4f1b92f6aea15e443ce1dc513a2bbb0df8bd9fb0a8ae5c203a6bf8a9f"} Nov 01 01:07:32.932267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.932258 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8" Nov 01 01:07:32.932524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.932287 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4ad6d4f1b92f6aea15e443ce1dc513a2bbb0df8bd9fb0a8ae5c203a6bf8a9f" Nov 01 01:07:32.933805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.933782 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" event={"ID":"5ee42edb-342d-478a-a090-368586987ac9","Type":"ContainerDied","Data":"5ec7f374814a9190fc0946fc7ea3c95a529953ccaec3e829508286c2fdba8cb0"} Nov 01 01:07:32.933943 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.933829 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn" Nov 01 01:07:32.933943 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:32.933831 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec7f374814a9190fc0946fc7ea3c95a529953ccaec3e829508286c2fdba8cb0" Nov 01 01:07:35.171347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.171300 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" containerID="cri-o://67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1" gracePeriod=45 Nov 01 01:07:35.458195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.458164 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 01:07:35.521100 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521069 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tmp-dir\") pod \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " Nov 01 01:07:35.521268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521143 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-model-cache\") pod \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " Nov 01 01:07:35.521268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521182 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tls-certs\" (UniqueName: \"kubernetes.io/secret/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tls-certs\") pod \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " Nov 01 01:07:35.521268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521249 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-dshm\") pod \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " Nov 01 01:07:35.521410 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521342 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tmp-dir" pod "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" (UID: "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.521410 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521380 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-home\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-home\") pod \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " Nov 01 01:07:35.521533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521418 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-model-cache" pod "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" (UID: "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.521533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521424 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-kube-api-access-db8n6\" (UniqueName: \"kubernetes.io/projected/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-kube-api-access-db8n6\") pod \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\" (UID: \"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326\") " Nov 01 01:07:35.521684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521663 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-home" pod "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" (UID: "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.521957 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521932 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.521957 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521956 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.522317 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.521966 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.523879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.523858 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-dshm" pod "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" (UID: "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:35.523999 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.523980 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-kube-api-access-db8n6" pod "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" (UID: "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326"). InnerVolumeSpecName "kube-api-access-db8n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:35.523999 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.523983 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tls-certs" pod "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" (UID: "50e2a7d3-54ff-4745-8d7e-2ba23a1a7326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:35.623246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.623208 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.623246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.623246 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.623502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.623265 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db8n6\" (UniqueName: \"kubernetes.io/projected/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326-kube-api-access-db8n6\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:35.960083 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.960052 4910 generic.go:356] "Generic (PLEG): container finished" podID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerID="67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1" exitCode=0 Nov 01 01:07:35.960271 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.960095 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" Nov 01 01:07:35.960271 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.960137 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" event={"ID":"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326","Type":"ContainerDied","Data":"67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1"} Nov 01 01:07:35.960271 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.960193 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl" event={"ID":"50e2a7d3-54ff-4745-8d7e-2ba23a1a7326","Type":"ContainerDied","Data":"e4bb04893e42432758aae7f46f58aaf20104ada3414e9ef444544b116ad91395"} Nov 01 01:07:35.960271 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.960224 4910 scope.go:122] "RemoveContainer" containerID="67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1" Nov 01 01:07:35.973009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.972985 4910 scope.go:122] "RemoveContainer" containerID="67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1" Nov 01 01:07:35.973281 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:07:35.973261 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1\": container with ID starting with 67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1 not found: ID does not exist" containerID="67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1" Nov 01 01:07:35.973354 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:35.973289 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1"} err="failed to get container status \"67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1\": rpc error: code = NotFound desc = could not find container \"67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1\": container with ID starting with 67a8c6774bc4efcaf701e07b13fe56592b3862b0620be5e61aea69eb54fbb3a1 not found: ID does not exist" Nov 01 01:07:36.126771 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:36.126739 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl"] Nov 01 01:07:36.130411 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:36.130398 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-948e22/simulator-w5-948e22-kserve-b8dfbd98b-c7hgl"] Nov 01 01:07:36.192022 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:36.191992 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" path="/var/lib/kubelet/pods/50e2a7d3-54ff-4745-8d7e-2ba23a1a7326/volumes" Nov 01 01:07:36.217891 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:36.217829 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt"] Nov 01 01:07:43.103098 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:07:43.103007 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:07:47.575646 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:47.574924 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr"] Nov 01 01:07:47.608815 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:47.608774 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f73fb0-78tbr"] Nov 01 01:07:48.193142 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:48.193108 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9235253a-2fac-4fa5-89ef-f478c11c5fca" path="/var/lib/kubelet/pods/9235253a-2fac-4fa5-89ef-f478c11c5fca/volumes" Nov 01 01:07:51.239342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.239296 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" podUID="d7253215-068b-424e-be32-4a7654cf124b" containerName="main" containerID="cri-o://8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1" gracePeriod=45 Nov 01 01:07:51.517788 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.517763 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 01:07:51.599246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599220 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-home\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-home\") pod \"d7253215-068b-424e-be32-4a7654cf124b\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " Nov 01 01:07:51.599434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599295 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7253215-068b-424e-be32-4a7654cf124b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7253215-068b-424e-be32-4a7654cf124b-tls-certs\") pod \"d7253215-068b-424e-be32-4a7654cf124b\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " Nov 01 01:07:51.599434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599331 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-dshm\") pod \"d7253215-068b-424e-be32-4a7654cf124b\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " Nov 01 01:07:51.599434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599370 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-tmp-dir\") pod \"d7253215-068b-424e-be32-4a7654cf124b\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " Nov 01 01:07:51.599574 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599462 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-model-cache\") pod \"d7253215-068b-424e-be32-4a7654cf124b\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " Nov 01 01:07:51.599685 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599560 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-home" pod "d7253215-068b-424e-be32-4a7654cf124b" (UID: "d7253215-068b-424e-be32-4a7654cf124b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:51.599749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599699 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7253215-068b-424e-be32-4a7654cf124b-kube-api-access-dlg6n\" (UniqueName: \"kubernetes.io/projected/d7253215-068b-424e-be32-4a7654cf124b-kube-api-access-dlg6n\") pod \"d7253215-068b-424e-be32-4a7654cf124b\" (UID: \"d7253215-068b-424e-be32-4a7654cf124b\") " Nov 01 01:07:51.599749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599703 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-model-cache" pod "d7253215-068b-424e-be32-4a7654cf124b" (UID: "d7253215-068b-424e-be32-4a7654cf124b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:51.599749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.599712 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-tmp-dir" pod "d7253215-068b-424e-be32-4a7654cf124b" (UID: "d7253215-068b-424e-be32-4a7654cf124b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:51.600219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.600199 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:51.600219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.600221 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:51.600405 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.600233 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:51.601925 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.601905 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-dshm" pod "d7253215-068b-424e-be32-4a7654cf124b" (UID: "d7253215-068b-424e-be32-4a7654cf124b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:07:51.602298 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.602278 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7253215-068b-424e-be32-4a7654cf124b-tls-certs" pod "d7253215-068b-424e-be32-4a7654cf124b" (UID: "d7253215-068b-424e-be32-4a7654cf124b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:51.602349 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.602304 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7253215-068b-424e-be32-4a7654cf124b-kube-api-access-dlg6n" pod "d7253215-068b-424e-be32-4a7654cf124b" (UID: "d7253215-068b-424e-be32-4a7654cf124b"). InnerVolumeSpecName "kube-api-access-dlg6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:51.701859 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.701831 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7253215-068b-424e-be32-4a7654cf124b-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:51.701859 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.701857 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7253215-068b-424e-be32-4a7654cf124b-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:51.702075 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:51.701871 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlg6n\" (UniqueName: \"kubernetes.io/projected/d7253215-068b-424e-be32-4a7654cf124b-kube-api-access-dlg6n\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:52.118572 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.118541 4910 generic.go:356] "Generic (PLEG): container finished" podID="d7253215-068b-424e-be32-4a7654cf124b" containerID="8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1" exitCode=0 Nov 01 01:07:52.118773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.118588 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" Nov 01 01:07:52.118773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.118654 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" event={"ID":"d7253215-068b-424e-be32-4a7654cf124b","Type":"ContainerDied","Data":"8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1"} Nov 01 01:07:52.118773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.118703 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt" event={"ID":"d7253215-068b-424e-be32-4a7654cf124b","Type":"ContainerDied","Data":"a4df6a9de53579a75503256fcba7bfab384009a9b7495f87f7b9736f7c8cd6ae"} Nov 01 01:07:52.118773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.118721 4910 scope.go:122] "RemoveContainer" containerID="8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1" Nov 01 01:07:52.132359 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.132338 4910 scope.go:122] "RemoveContainer" containerID="8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1" Nov 01 01:07:52.132665 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:07:52.132645 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1\": container with ID starting with 8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1 not found: ID does not exist" containerID="8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1" Nov 01 01:07:52.132717 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.132674 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1"} err="failed to get container status \"8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1\": rpc error: code = NotFound desc = could not find container \"8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1\": container with ID starting with 8c5320830c6dd7ca0836995148f5e5b2b421942c5bcea7e4871d9b9030240bd1 not found: ID does not exist" Nov 01 01:07:52.285583 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.285552 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt"] Nov 01 01:07:52.335651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:52.335609 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b112c158/e2e-lifecycle-model-b112c158-kserve-5d969c989c-tzkzt"] Nov 01 01:07:54.192076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:54.192038 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7253215-068b-424e-be32-4a7654cf124b" path="/var/lib/kubelet/pods/d7253215-068b-424e-be32-4a7654cf124b/volumes" Nov 01 01:07:54.761617 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:54.761586 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw"] Nov 01 01:07:54.761870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:54.761845 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" containerID="cri-o://212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe" gracePeriod=30 Nov 01 01:07:55.955949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.955918 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg"] Nov 01 01:07:55.956485 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956472 4910 cpu_manager.go:414] "containerMap: removing container" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" Nov 01 01:07:55.956540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956486 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" Nov 01 01:07:55.956540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956502 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1982ee14-47f4-4a02-9040-6b1200ca9864" containerName="revoke-keys" Nov 01 01:07:55.956540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956507 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1982ee14-47f4-4a02-9040-6b1200ca9864" containerName="revoke-keys" Nov 01 01:07:55.956540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956522 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5ee42edb-342d-478a-a090-368586987ac9" containerName="revoke-keys" Nov 01 01:07:55.956540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956527 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee42edb-342d-478a-a090-368586987ac9" containerName="revoke-keys" Nov 01 01:07:55.956540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956540 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d7253215-068b-424e-be32-4a7654cf124b" containerName="main" Nov 01 01:07:55.956770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956545 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7253215-068b-424e-be32-4a7654cf124b" containerName="main" Nov 01 01:07:55.956770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956644 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee42edb-342d-478a-a090-368586987ac9" containerName="revoke-keys" Nov 01 01:07:55.956770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956652 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7253215-068b-424e-be32-4a7654cf124b" containerName="main" Nov 01 01:07:55.956770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956664 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e2a7d3-54ff-4745-8d7e-2ba23a1a7326" containerName="main" Nov 01 01:07:55.956770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.956671 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1982ee14-47f4-4a02-9040-6b1200ca9864" containerName="revoke-keys" Nov 01 01:07:55.959942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:55.959925 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.022995 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.022965 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg"] Nov 01 01:07:56.053473 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.053442 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hvn\" (UniqueName: \"kubernetes.io/projected/284d5886-0fc2-4c05-99e8-11b5319419be-kube-api-access-f4hvn\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.054041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.053987 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.054250 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.054164 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.063585 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.063564 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:07:56.063687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.063680 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:07:56.155415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.155379 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.155579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.155471 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.155713 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.155598 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hvn\" (UniqueName: \"kubernetes.io/projected/284d5886-0fc2-4c05-99e8-11b5319419be-kube-api-access-f4hvn\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.156800 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.156771 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.158257 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.158231 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.216853 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.216790 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hvn\" (UniqueName: \"kubernetes.io/projected/284d5886-0fc2-4c05-99e8-11b5319419be-kube-api-access-f4hvn\") pod \"maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.274158 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.274129 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:56.690115 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:56.690081 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg"] Nov 01 01:07:57.169698 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:57.169657 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" event={"ID":"284d5886-0fc2-4c05-99e8-11b5319419be","Type":"ContainerStarted","Data":"c5d9efc6cd6ab2973513396329537e711596856461b4a790869fdc3a0689239f"} Nov 01 01:07:57.170099 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:57.169712 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" event={"ID":"284d5886-0fc2-4c05-99e8-11b5319419be","Type":"ContainerStarted","Data":"4dc83b37f7f8d0bd1939487b48f8d76aaf17a0c1694323c7b34d51f424a66a81"} Nov 01 01:07:58.279407 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:58.279371 4910 generic.go:356] "Generic (PLEG): container finished" podID="284d5886-0fc2-4c05-99e8-11b5319419be" containerID="c5d9efc6cd6ab2973513396329537e711596856461b4a790869fdc3a0689239f" exitCode=0 Nov 01 01:07:58.366348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:58.347879 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" event={"ID":"284d5886-0fc2-4c05-99e8-11b5319419be","Type":"ContainerDied","Data":"c5d9efc6cd6ab2973513396329537e711596856461b4a790869fdc3a0689239f"} Nov 01 01:07:59.453223 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.453194 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:07:59.522934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.522904 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-openshift-ca\") pod \"284d5886-0fc2-4c05-99e8-11b5319419be\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " Nov 01 01:07:59.523105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.523037 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284d5886-0fc2-4c05-99e8-11b5319419be-kube-api-access-f4hvn\" (UniqueName: \"kubernetes.io/projected/284d5886-0fc2-4c05-99e8-11b5319419be-kube-api-access-f4hvn\") pod \"284d5886-0fc2-4c05-99e8-11b5319419be\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " Nov 01 01:07:59.523153 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.523121 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-xks-ca\") pod \"284d5886-0fc2-4c05-99e8-11b5319419be\" (UID: \"284d5886-0fc2-4c05-99e8-11b5319419be\") " Nov 01 01:07:59.523299 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.523279 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-openshift-ca" pod "284d5886-0fc2-4c05-99e8-11b5319419be" (UID: "284d5886-0fc2-4c05-99e8-11b5319419be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:07:59.523603 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.523579 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:59.525942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.525919 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-xks-ca" pod "284d5886-0fc2-4c05-99e8-11b5319419be" (UID: "284d5886-0fc2-4c05-99e8-11b5319419be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:07:59.526045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.526017 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284d5886-0fc2-4c05-99e8-11b5319419be-kube-api-access-f4hvn" pod "284d5886-0fc2-4c05-99e8-11b5319419be" (UID: "284d5886-0fc2-4c05-99e8-11b5319419be"). InnerVolumeSpecName "kube-api-access-f4hvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:07:59.627988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.627900 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hvn\" (UniqueName: \"kubernetes.io/projected/284d5886-0fc2-4c05-99e8-11b5319419be-kube-api-access-f4hvn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:59.627988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.627937 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284d5886-0fc2-4c05-99e8-11b5319419be-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:07:59.838882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.838830 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz"] Nov 01 01:07:59.842780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:07:59.842754 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4143b7-kzglz"] Nov 01 01:08:00.024025 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.024003 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:08:00.133730 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.133695 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-data\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.133730 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.133736 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-credential-socket\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134031 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.133823 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-kube-api-access-ch8fk\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-kube-api-access-ch8fk\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134031 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.133859 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-podinfo\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134031 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.133881 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istiod-ca-cert\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134031 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.133972 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-data" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:00.134031 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.133980 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-socket\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134066 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-token\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134109 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-certs\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134108 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-credential-socket" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:00.134334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134137 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-envoy\") pod \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\" (UID: \"c8b7ff3e-84d4-4241-81b0-5648a120a61a\") " Nov 01 01:08:00.134334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134324 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-socket" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:00.134667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134340 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-certs" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:00.134667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134576 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istiod-ca-cert" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:00.135028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.134985 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.135028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.135013 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.135193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.135029 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.135193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.135045 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.135193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.135059 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.136401 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.136364 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-kube-api-access-ch8fk" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "kube-api-access-ch8fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:00.136717 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.136690 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-token" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:00.136846 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.136816 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-podinfo" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:08:00.137017 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.136994 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-envoy" pod "c8b7ff3e-84d4-4241-81b0-5648a120a61a" (UID: "c8b7ff3e-84d4-4241-81b0-5648a120a61a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:00.205393 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.205314 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0438b1-c83c-42d1-87f2-b68235f1ef2d" path="/var/lib/kubelet/pods/ae0438b1-c83c-42d1-87f2-b68235f1ef2d/volumes" Nov 01 01:08:00.235733 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.235702 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8fk\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-kube-api-access-ch8fk\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.235733 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.235728 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.235893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.235738 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.235893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.235747 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b7ff3e-84d4-4241-81b0-5648a120a61a-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:00.301696 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.301651 4910 generic.go:356] "Generic (PLEG): container finished" podID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerID="212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe" exitCode=0 Nov 01 01:08:00.301903 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.301714 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" Nov 01 01:08:00.301903 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.301744 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" event={"ID":"c8b7ff3e-84d4-4241-81b0-5648a120a61a","Type":"ContainerDied","Data":"212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe"} Nov 01 01:08:00.301903 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.301846 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw" event={"ID":"c8b7ff3e-84d4-4241-81b0-5648a120a61a","Type":"ContainerDied","Data":"e62ef410fdfddefa4ab8821e831039f5a183878a861127efa0ad0a08394c5add"} Nov 01 01:08:00.301903 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.301876 4910 scope.go:122] "RemoveContainer" containerID="212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe" Nov 01 01:08:00.303752 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.303726 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" event={"ID":"284d5886-0fc2-4c05-99e8-11b5319419be","Type":"ContainerDied","Data":"4dc83b37f7f8d0bd1939487b48f8d76aaf17a0c1694323c7b34d51f424a66a81"} Nov 01 01:08:00.303872 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.303773 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc83b37f7f8d0bd1939487b48f8d76aaf17a0c1694323c7b34d51f424a66a81" Nov 01 01:08:00.303872 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.303787 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg" Nov 01 01:08:00.314963 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.314940 4910 scope.go:122] "RemoveContainer" containerID="8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8" Nov 01 01:08:00.326139 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.326120 4910 scope.go:122] "RemoveContainer" containerID="212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe" Nov 01 01:08:00.326400 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:00.326378 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe\": container with ID starting with 212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe not found: ID does not exist" containerID="212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe" Nov 01 01:08:00.326449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.326412 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe"} err="failed to get container status \"212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe\": rpc error: code = NotFound desc = could not find container \"212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe\": container with ID starting with 212f210822ce4f13f0aa969d59deda3dd0c46b2fa34e3e88a0a97c924dadc2fe not found: ID does not exist" Nov 01 01:08:00.326449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.326436 4910 scope.go:122] "RemoveContainer" containerID="8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8" Nov 01 01:08:00.326661 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:00.326614 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8\": container with ID starting with 8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8 not found: ID does not exist" containerID="8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8" Nov 01 01:08:00.326761 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.326658 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8"} err="failed to get container status \"8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8\": rpc error: code = NotFound desc = could not find container \"8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8\": container with ID starting with 8c42ac494d1473ff403334056f11bf45cf2e0e2734aefe8875a892ff9ef6c9d8 not found: ID does not exist" Nov 01 01:08:00.347579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.347551 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742"] Nov 01 01:08:00.494891 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.494794 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw"] Nov 01 01:08:00.498959 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:00.498927 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f73fb0-openshift-default-6dd5699c75-kxjdw"] Nov 01 01:08:02.194912 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:02.194881 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" path="/var/lib/kubelet/pods/c8b7ff3e-84d4-4241-81b0-5648a120a61a/volumes" Nov 01 01:08:06.883342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:06.883297 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk"] Nov 01 01:08:06.883926 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:06.883552 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" containerID="cri-o://938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d" gracePeriod=30 Nov 01 01:08:07.505663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:07.505631 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222"] Nov 01 01:08:07.617800 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:07.617767 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8"] Nov 01 01:08:09.603590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.603556 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z"] Nov 01 01:08:09.604135 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.604120 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" Nov 01 01:08:09.604177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.604135 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" Nov 01 01:08:09.604177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.604145 4910 cpu_manager.go:414] "containerMap: removing container" podUID="284d5886-0fc2-4c05-99e8-11b5319419be" containerName="revoke-keys" Nov 01 01:08:09.604177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.604150 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284d5886-0fc2-4c05-99e8-11b5319419be" containerName="revoke-keys" Nov 01 01:08:09.604268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.604244 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" Nov 01 01:08:09.604268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.604258 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="284d5886-0fc2-4c05-99e8-11b5319419be" containerName="revoke-keys" Nov 01 01:08:09.609454 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.609439 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.658185 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.658150 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km"] Nov 01 01:08:09.658774 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.658755 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" Nov 01 01:08:09.658774 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.658774 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" Nov 01 01:08:09.658940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.658908 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b7ff3e-84d4-4241-81b0-5648a120a61a" containerName="istio-proxy" Nov 01 01:08:09.662423 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.662405 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.663978 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.663953 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z"] Nov 01 01:08:09.678485 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.678463 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrgx\" (UniqueName: \"kubernetes.io/projected/7cb2adde-1af6-4023-962c-1f5376d964f5-kube-api-access-qrrgx\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.678579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.678506 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.678643 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.678583 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.704373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.704352 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:08:09.704509 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.704428 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:08:09.721141 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.721102 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km"] Nov 01 01:08:09.780236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.780204 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.780426 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.780257 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29fw\" (UniqueName: \"kubernetes.io/projected/457aa14d-cf8a-439f-8048-1f50d4431442-kube-api-access-q29fw\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.780426 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.780300 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.780554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.780429 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.780554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.780462 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrgx\" (UniqueName: \"kubernetes.io/projected/7cb2adde-1af6-4023-962c-1f5376d964f5-kube-api-access-qrrgx\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.780554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.780525 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.781045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.781019 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.782922 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.782899 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.841079 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.841055 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrgx\" (UniqueName: \"kubernetes.io/projected/7cb2adde-1af6-4023-962c-1f5376d964f5-kube-api-access-qrrgx\") pod \"maas-api-revoke-sub-31a639c6-stm7z\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.882043 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.881974 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.882170 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.882093 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.882267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.882250 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29fw\" (UniqueName: \"kubernetes.io/projected/457aa14d-cf8a-439f-8048-1f50d4431442-kube-api-access-q29fw\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.882667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.882645 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.884433 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.884412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.923561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.923537 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:09.943190 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.943171 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29fw\" (UniqueName: \"kubernetes.io/projected/457aa14d-cf8a-439f-8048-1f50d4431442-kube-api-access-q29fw\") pod \"maas-api-revoke-sub-8bcf8c35-fd7km\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:09.976477 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:09.976242 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:10.307969 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:10.307937 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z"] Nov 01 01:08:10.308338 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:08:10.308312 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb2adde_1af6_4023_962c_1f5376d964f5.slice/crio-8d5ec9fe2a1cee64f3e3057b81d3735fb4dbbaff8838a7f774b2eecd7597908c WatchSource:0}: Error finding container 8d5ec9fe2a1cee64f3e3057b81d3735fb4dbbaff8838a7f774b2eecd7597908c: Status 404 returned error can't find the container with id 8d5ec9fe2a1cee64f3e3057b81d3735fb4dbbaff8838a7f774b2eecd7597908c Nov 01 01:08:10.386654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:10.386590 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" event={"ID":"7cb2adde-1af6-4023-962c-1f5376d964f5","Type":"ContainerStarted","Data":"8d5ec9fe2a1cee64f3e3057b81d3735fb4dbbaff8838a7f774b2eecd7597908c"} Nov 01 01:08:10.419356 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:10.419332 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km"] Nov 01 01:08:10.420337 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:08:10.420313 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457aa14d_cf8a_439f_8048_1f50d4431442.slice/crio-f0e836adfc8089b7e3051f8ca7701c63ca4223dbcb3236a76d11ae3d44975f81 WatchSource:0}: Error finding container f0e836adfc8089b7e3051f8ca7701c63ca4223dbcb3236a76d11ae3d44975f81: Status 404 returned error can't find the container with id f0e836adfc8089b7e3051f8ca7701c63ca4223dbcb3236a76d11ae3d44975f81 Nov 01 01:08:11.396209 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:11.396178 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" event={"ID":"457aa14d-cf8a-439f-8048-1f50d4431442","Type":"ContainerStarted","Data":"81180813f2969fd98049d6b79ce2b72700654f1dc989d46a3a1d76da84132ac9"} Nov 01 01:08:11.396697 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:11.396227 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" event={"ID":"457aa14d-cf8a-439f-8048-1f50d4431442","Type":"ContainerStarted","Data":"f0e836adfc8089b7e3051f8ca7701c63ca4223dbcb3236a76d11ae3d44975f81"} Nov 01 01:08:11.397921 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:11.397895 4910 generic.go:356] "Generic (PLEG): container finished" podID="7cb2adde-1af6-4023-962c-1f5376d964f5" containerID="841a8aebb96ddfe51ae86032fd339d16cec3f2396bf36738d6bbddcb4f25e5f0" exitCode=0 Nov 01 01:08:11.398020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:11.397978 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" event={"ID":"7cb2adde-1af6-4023-962c-1f5376d964f5","Type":"ContainerDied","Data":"841a8aebb96ddfe51ae86032fd339d16cec3f2396bf36738d6bbddcb4f25e5f0"} Nov 01 01:08:11.515670 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:11.511596 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" podStartSLOduration=2.5115791229999997 podStartE2EDuration="2.511579123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:08:11.511045684 +0000 UTC m=+2681.490628022" watchObservedRunningTime="2026-11-01 01:08:11.511579123 +0000 UTC m=+2681.491161460" Nov 01 01:08:11.971089 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:11.971055 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 01 01:08:11.971237 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:11.971117 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 01 01:08:12.149188 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.149127 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 01:08:12.309788 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.309751 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-socket\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.309983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.309804 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-token\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-token\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.309983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.309838 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-data\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.309983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.309914 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-credential-socket\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.310245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310052 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-socket" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.310245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310077 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-certs\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.310245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310121 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-podinfo\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.310245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310155 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-kube-api-access-jfrx6\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-kube-api-access-jfrx6\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.310245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310208 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a66a46-af99-4275-9ed6-1c513dd1f327-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a66a46-af99-4275-9ed6-1c513dd1f327-istiod-ca-cert\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.310519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310259 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-envoy\") pod \"e0a66a46-af99-4275-9ed6-1c513dd1f327\" (UID: \"e0a66a46-af99-4275-9ed6-1c513dd1f327\") " Nov 01 01:08:12.310519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310281 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-data" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.310519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310291 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-credential-socket" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.310519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310349 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-certs" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.310764 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.310668 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a66a46-af99-4275-9ed6-1c513dd1f327-istiod-ca-cert" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:12.311681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.311437 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.311681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.311549 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a66a46-af99-4275-9ed6-1c513dd1f327-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.311681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.311575 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.311681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.311590 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.311681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.311606 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.313154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.313130 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-podinfo" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:08:12.314481 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.314457 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-kube-api-access-jfrx6" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "kube-api-access-jfrx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:12.314594 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.314575 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-envoy" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:12.314709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.314691 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-token" pod "e0a66a46-af99-4275-9ed6-1c513dd1f327" (UID: "e0a66a46-af99-4275-9ed6-1c513dd1f327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:12.410136 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.410047 4910 generic.go:356] "Generic (PLEG): container finished" podID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerID="938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d" exitCode=0 Nov 01 01:08:12.410136 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.410115 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" Nov 01 01:08:12.410707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.410136 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" event={"ID":"e0a66a46-af99-4275-9ed6-1c513dd1f327","Type":"ContainerDied","Data":"938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d"} Nov 01 01:08:12.410707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.410216 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk" event={"ID":"e0a66a46-af99-4275-9ed6-1c513dd1f327","Type":"ContainerDied","Data":"3f2ec702dcd9ca01ff878108f4b80012808f4ba1355945d6e65e1e91bd73397b"} Nov 01 01:08:12.410707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.410248 4910 scope.go:122] "RemoveContainer" containerID="938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d" Nov 01 01:08:12.413214 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.413192 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.413381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.413356 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfrx6\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-kube-api-access-jfrx6\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.413518 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.413385 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.413518 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.413402 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a66a46-af99-4275-9ed6-1c513dd1f327-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.413518 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.413320 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" event={"ID":"457aa14d-cf8a-439f-8048-1f50d4431442","Type":"ContainerDied","Data":"81180813f2969fd98049d6b79ce2b72700654f1dc989d46a3a1d76da84132ac9"} Nov 01 01:08:12.413518 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.413298 4910 generic.go:356] "Generic (PLEG): container finished" podID="457aa14d-cf8a-439f-8048-1f50d4431442" containerID="81180813f2969fd98049d6b79ce2b72700654f1dc989d46a3a1d76da84132ac9" exitCode=0 Nov 01 01:08:12.426047 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.425051 4910 scope.go:122] "RemoveContainer" containerID="938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d" Nov 01 01:08:12.426047 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:12.425386 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d\": container with ID starting with 938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d not found: ID does not exist" containerID="938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d" Nov 01 01:08:12.426047 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.425421 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d"} err="failed to get container status \"938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d\": rpc error: code = NotFound desc = could not find container \"938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d\": container with ID starting with 938fddd31d2c854cefbdd287ad56faaab4e64d839be19f314edf756f2b32a84d not found: ID does not exist" Nov 01 01:08:12.535766 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.535707 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" podUID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" containerName="payload-processing" containerID="cri-o://97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced" gracePeriod=25 Nov 01 01:08:12.581752 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.581723 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk"] Nov 01 01:08:12.587567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.587535 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4143b7-openshift-default-7894568887-d2hjk"] Nov 01 01:08:12.608598 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.608570 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:12.618882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.618847 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerName="payload-pre-processing" containerID="cri-o://408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7" gracePeriod=25 Nov 01 01:08:12.721070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.720992 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-openshift-ca\") pod \"7cb2adde-1af6-4023-962c-1f5376d964f5\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " Nov 01 01:08:12.721070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.721055 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-xks-ca\") pod \"7cb2adde-1af6-4023-962c-1f5376d964f5\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " Nov 01 01:08:12.721311 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.721177 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb2adde-1af6-4023-962c-1f5376d964f5-kube-api-access-qrrgx\" (UniqueName: \"kubernetes.io/projected/7cb2adde-1af6-4023-962c-1f5376d964f5-kube-api-access-qrrgx\") pod \"7cb2adde-1af6-4023-962c-1f5376d964f5\" (UID: \"7cb2adde-1af6-4023-962c-1f5376d964f5\") " Nov 01 01:08:12.721418 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.721391 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-openshift-ca" pod "7cb2adde-1af6-4023-962c-1f5376d964f5" (UID: "7cb2adde-1af6-4023-962c-1f5376d964f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:12.721664 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.721612 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.723497 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.723470 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb2adde-1af6-4023-962c-1f5376d964f5-kube-api-access-qrrgx" pod "7cb2adde-1af6-4023-962c-1f5376d964f5" (UID: "7cb2adde-1af6-4023-962c-1f5376d964f5"). InnerVolumeSpecName "kube-api-access-qrrgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:12.723606 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.723545 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-xks-ca" pod "7cb2adde-1af6-4023-962c-1f5376d964f5" (UID: "7cb2adde-1af6-4023-962c-1f5376d964f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:08:12.829275 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.829241 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2adde-1af6-4023-962c-1f5376d964f5-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:12.829275 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:12.829276 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrgx\" (UniqueName: \"kubernetes.io/projected/7cb2adde-1af6-4023-962c-1f5376d964f5-kube-api-access-qrrgx\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:13.104112 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:13.104076 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:08:13.423776 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.423684 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" event={"ID":"7cb2adde-1af6-4023-962c-1f5376d964f5","Type":"ContainerDied","Data":"8d5ec9fe2a1cee64f3e3057b81d3735fb4dbbaff8838a7f774b2eecd7597908c"} Nov 01 01:08:13.423776 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.423744 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d5ec9fe2a1cee64f3e3057b81d3735fb4dbbaff8838a7f774b2eecd7597908c" Nov 01 01:08:13.423776 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.423696 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z" Nov 01 01:08:13.585204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.585176 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:13.740789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.740712 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457aa14d-cf8a-439f-8048-1f50d4431442-kube-api-access-q29fw\" (UniqueName: \"kubernetes.io/projected/457aa14d-cf8a-439f-8048-1f50d4431442-kube-api-access-q29fw\") pod \"457aa14d-cf8a-439f-8048-1f50d4431442\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " Nov 01 01:08:13.740956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.740862 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-xks-ca\") pod \"457aa14d-cf8a-439f-8048-1f50d4431442\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " Nov 01 01:08:13.740956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.740937 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-openshift-ca\") pod \"457aa14d-cf8a-439f-8048-1f50d4431442\" (UID: \"457aa14d-cf8a-439f-8048-1f50d4431442\") " Nov 01 01:08:13.741688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.741657 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-openshift-ca" pod "457aa14d-cf8a-439f-8048-1f50d4431442" (UID: "457aa14d-cf8a-439f-8048-1f50d4431442"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:13.743245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.743216 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457aa14d-cf8a-439f-8048-1f50d4431442-kube-api-access-q29fw" pod "457aa14d-cf8a-439f-8048-1f50d4431442" (UID: "457aa14d-cf8a-439f-8048-1f50d4431442"). InnerVolumeSpecName "kube-api-access-q29fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:13.743717 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.743694 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-xks-ca" pod "457aa14d-cf8a-439f-8048-1f50d4431442" (UID: "457aa14d-cf8a-439f-8048-1f50d4431442"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:08:13.843132 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.843102 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:13.843132 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.843129 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457aa14d-cf8a-439f-8048-1f50d4431442-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:13.843380 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:13.843138 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29fw\" (UniqueName: \"kubernetes.io/projected/457aa14d-cf8a-439f-8048-1f50d4431442-kube-api-access-q29fw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:14.192389 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:14.192354 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" path="/var/lib/kubelet/pods/e0a66a46-af99-4275-9ed6-1c513dd1f327/volumes" Nov 01 01:08:14.434290 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:14.434244 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" event={"ID":"457aa14d-cf8a-439f-8048-1f50d4431442","Type":"ContainerDied","Data":"f0e836adfc8089b7e3051f8ca7701c63ca4223dbcb3236a76d11ae3d44975f81"} Nov 01 01:08:14.434721 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:14.434308 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e836adfc8089b7e3051f8ca7701c63ca4223dbcb3236a76d11ae3d44975f81" Nov 01 01:08:14.434721 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:14.434260 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km" Nov 01 01:08:15.372058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.371986 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" containerID="cri-o://6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12" gracePeriod=45 Nov 01 01:08:15.415334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.415299 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": dial tcp 10.132.0.98:8000: connect: connection refused" Nov 01 01:08:15.640856 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.640795 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 01:08:15.771818 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.771786 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tmp-dir\") pod \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " Nov 01 01:08:15.772003 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.771975 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-model-cache\") pod \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " Nov 01 01:08:15.772083 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772047 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-home\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-home\") pod \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " Nov 01 01:08:15.772145 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772084 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tls-certs\") pod \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " Nov 01 01:08:15.772145 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772081 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tmp-dir" pod "e9b5e5da-310d-469f-b7b3-cd6f08314deb" (UID: "e9b5e5da-310d-469f-b7b3-cd6f08314deb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.772259 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772151 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b5e5da-310d-469f-b7b3-cd6f08314deb-kube-api-access-52dv9\" (UniqueName: \"kubernetes.io/projected/e9b5e5da-310d-469f-b7b3-cd6f08314deb-kube-api-access-52dv9\") pod \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " Nov 01 01:08:15.772321 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772258 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-dshm\") pod \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\" (UID: \"e9b5e5da-310d-469f-b7b3-cd6f08314deb\") " Nov 01 01:08:15.772701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772673 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-model-cache" pod "e9b5e5da-310d-469f-b7b3-cd6f08314deb" (UID: "e9b5e5da-310d-469f-b7b3-cd6f08314deb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.772838 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772817 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.772909 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772845 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.772978 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.772909 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-home" pod "e9b5e5da-310d-469f-b7b3-cd6f08314deb" (UID: "e9b5e5da-310d-469f-b7b3-cd6f08314deb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.775980 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.775938 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b5e5da-310d-469f-b7b3-cd6f08314deb-kube-api-access-52dv9" pod "e9b5e5da-310d-469f-b7b3-cd6f08314deb" (UID: "e9b5e5da-310d-469f-b7b3-cd6f08314deb"). InnerVolumeSpecName "kube-api-access-52dv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:15.775980 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.775926 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tls-certs" pod "e9b5e5da-310d-469f-b7b3-cd6f08314deb" (UID: "e9b5e5da-310d-469f-b7b3-cd6f08314deb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:08:15.776571 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.776548 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-dshm" pod "e9b5e5da-310d-469f-b7b3-cd6f08314deb" (UID: "e9b5e5da-310d-469f-b7b3-cd6f08314deb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:15.874517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.874486 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.874517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.874514 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b5e5da-310d-469f-b7b3-cd6f08314deb-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.874517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.874525 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52dv9\" (UniqueName: \"kubernetes.io/projected/e9b5e5da-310d-469f-b7b3-cd6f08314deb-kube-api-access-52dv9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:15.874798 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:15.874534 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b5e5da-310d-469f-b7b3-cd6f08314deb-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:16.451108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.451027 4910 generic.go:356] "Generic (PLEG): container finished" podID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerID="6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12" exitCode=0 Nov 01 01:08:16.451108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.451075 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" Nov 01 01:08:16.451306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.451116 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" event={"ID":"e9b5e5da-310d-469f-b7b3-cd6f08314deb","Type":"ContainerDied","Data":"6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12"} Nov 01 01:08:16.451306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.451162 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742" event={"ID":"e9b5e5da-310d-469f-b7b3-cd6f08314deb","Type":"ContainerDied","Data":"80be61381a63bca8113ff198e7fd420e31f32f90757221422f6d682aa9c14e02"} Nov 01 01:08:16.451306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.451184 4910 scope.go:122] "RemoveContainer" containerID="6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12" Nov 01 01:08:16.461702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.461676 4910 scope.go:122] "RemoveContainer" containerID="6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12" Nov 01 01:08:16.462001 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:16.461978 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12\": container with ID starting with 6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12 not found: ID does not exist" containerID="6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12" Nov 01 01:08:16.462052 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.462010 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12"} err="failed to get container status \"6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12\": rpc error: code = NotFound desc = could not find container \"6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12\": container with ID starting with 6b4875f169f7399271599e441b2caf1306bbb1b26c98ab1beb6a1acd301c6b12 not found: ID does not exist" Nov 01 01:08:16.620613 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.620582 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742"] Nov 01 01:08:16.670056 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:16.670025 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-32439f/distinct-w0-32439f-kserve-76c8d8dc7d-v4742"] Nov 01 01:08:18.193910 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:18.193866 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" path="/var/lib/kubelet/pods/e9b5e5da-310d-469f-b7b3-cd6f08314deb/volumes" Nov 01 01:08:21.672618 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:21.672569 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:08:21.673174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:21.672577 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" podUID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:08:21.724218 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:21.724187 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q"] Nov 01 01:08:21.774008 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:21.773980 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-948e22-bgx9q"] Nov 01 01:08:22.192765 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:22.192724 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcaa2cb-297e-485d-b288-0707b1111692" path="/var/lib/kubelet/pods/9fcaa2cb-297e-485d-b288-0707b1111692/volumes" Nov 01 01:08:28.855874 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:28.855839 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx"] Nov 01 01:08:28.856346 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:28.856066 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" containerID="cri-o://32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613" gracePeriod=30 Nov 01 01:08:30.853207 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:30.849852 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:08:30.864427 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:30.864348 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:08:30.865497 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:30.865472 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:08:30.875491 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:30.875472 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:08:31.671704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:31.671664 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:08:31.671919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:31.671792 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" podUID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:08:32.490676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:32.490645 4910 scope.go:122] "RemoveContainer" containerID="e2f3ee4da2edec5de9b026df34599c597325030a0eb31b33bae20544e9b2b3ba" Nov 01 01:08:32.503677 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:32.503618 4910 scope.go:122] "RemoveContainer" containerID="805ed5a9bf5f98bf3282db60343e9d459c2795b85cab5acf197d02706d2b4ef8" Nov 01 01:08:32.513414 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:32.513393 4910 scope.go:122] "RemoveContainer" containerID="a73d514a69a68a4aeaee27b610d17694ec1e69090db206832a65ffab1ccc49c0" Nov 01 01:08:32.922012 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:32.921984 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 01:08:32.955255 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:32.955228 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 01:08:33.010474 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.010448 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0972708d-bace-41a7-a8f3-72d81578aeb2-kube-api-access-tp422\" (UniqueName: \"kubernetes.io/projected/0972708d-bace-41a7-a8f3-72d81578aeb2-kube-api-access-tp422\") pod \"0972708d-bace-41a7-a8f3-72d81578aeb2\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " Nov 01 01:08:33.010615 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.010581 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86075eb8-bef0-4de5-b920-a7d36dff0f3c-kube-api-access-ptfkt\" (UniqueName: \"kubernetes.io/projected/86075eb8-bef0-4de5-b920-a7d36dff0f3c-kube-api-access-ptfkt\") pod \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " Nov 01 01:08:33.010708 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.010694 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86075eb8-bef0-4de5-b920-a7d36dff0f3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86075eb8-bef0-4de5-b920-a7d36dff0f3c-plugins-config-volume\") pod \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\" (UID: \"86075eb8-bef0-4de5-b920-a7d36dff0f3c\") " Nov 01 01:08:33.010841 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.010826 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0972708d-bace-41a7-a8f3-72d81578aeb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0972708d-bace-41a7-a8f3-72d81578aeb2-plugins-config-volume\") pod \"0972708d-bace-41a7-a8f3-72d81578aeb2\" (UID: \"0972708d-bace-41a7-a8f3-72d81578aeb2\") " Nov 01 01:08:33.011126 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.011047 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86075eb8-bef0-4de5-b920-a7d36dff0f3c-plugins-config-volume" pod "86075eb8-bef0-4de5-b920-a7d36dff0f3c" (UID: "86075eb8-bef0-4de5-b920-a7d36dff0f3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:33.011408 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.011235 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86075eb8-bef0-4de5-b920-a7d36dff0f3c-plugins-config-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:33.011471 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.011412 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0972708d-bace-41a7-a8f3-72d81578aeb2-plugins-config-volume" pod "0972708d-bace-41a7-a8f3-72d81578aeb2" (UID: "0972708d-bace-41a7-a8f3-72d81578aeb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:33.012836 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.012812 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0972708d-bace-41a7-a8f3-72d81578aeb2-kube-api-access-tp422" pod "0972708d-bace-41a7-a8f3-72d81578aeb2" (UID: "0972708d-bace-41a7-a8f3-72d81578aeb2"). InnerVolumeSpecName "kube-api-access-tp422". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:33.013092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.013077 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86075eb8-bef0-4de5-b920-a7d36dff0f3c-kube-api-access-ptfkt" pod "86075eb8-bef0-4de5-b920-a7d36dff0f3c" (UID: "86075eb8-bef0-4de5-b920-a7d36dff0f3c"). InnerVolumeSpecName "kube-api-access-ptfkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:33.112690 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.112656 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptfkt\" (UniqueName: \"kubernetes.io/projected/86075eb8-bef0-4de5-b920-a7d36dff0f3c-kube-api-access-ptfkt\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:33.112690 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.112682 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0972708d-bace-41a7-a8f3-72d81578aeb2-plugins-config-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:33.112690 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.112692 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp422\" (UniqueName: \"kubernetes.io/projected/0972708d-bace-41a7-a8f3-72d81578aeb2-kube-api-access-tp422\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:33.668072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.668036 4910 generic.go:356] "Generic (PLEG): container finished" podID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerID="408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7" exitCode=0 Nov 01 01:08:33.668557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.668106 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" Nov 01 01:08:33.668557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.668114 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" event={"ID":"0972708d-bace-41a7-a8f3-72d81578aeb2","Type":"ContainerDied","Data":"408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7"} Nov 01 01:08:33.668557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.668179 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8" event={"ID":"0972708d-bace-41a7-a8f3-72d81578aeb2","Type":"ContainerDied","Data":"5976d275c8e5299c74a8560a2d36079b748d9a94723b99487777358a48bf7057"} Nov 01 01:08:33.668557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.668211 4910 scope.go:122] "RemoveContainer" containerID="408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7" Nov 01 01:08:33.669967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.669941 4910 generic.go:356] "Generic (PLEG): container finished" podID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" containerID="97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced" exitCode=0 Nov 01 01:08:33.670081 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.670054 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" event={"ID":"86075eb8-bef0-4de5-b920-a7d36dff0f3c","Type":"ContainerDied","Data":"97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced"} Nov 01 01:08:33.670149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.670101 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" event={"ID":"86075eb8-bef0-4de5-b920-a7d36dff0f3c","Type":"ContainerDied","Data":"16b6b1e2496631eac88909113b2713bef042ccd3157db596d00e620690e7e9c9"} Nov 01 01:08:33.670209 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.670190 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222" Nov 01 01:08:33.681855 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.681603 4910 scope.go:122] "RemoveContainer" containerID="408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7" Nov 01 01:08:33.681950 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:33.681920 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7\": container with ID starting with 408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7 not found: ID does not exist" containerID="408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7" Nov 01 01:08:33.682022 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.681949 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7"} err="failed to get container status \"408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7\": rpc error: code = NotFound desc = could not find container \"408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7\": container with ID starting with 408306d6797bc8e6a30aacdf0c4fb5441c10be49d4852dca503f862354aa47f7 not found: ID does not exist" Nov 01 01:08:33.682022 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.681971 4910 scope.go:122] "RemoveContainer" containerID="97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced" Nov 01 01:08:33.694187 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.694163 4910 scope.go:122] "RemoveContainer" containerID="97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced" Nov 01 01:08:33.694526 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:33.694496 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced\": container with ID starting with 97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced not found: ID does not exist" containerID="97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced" Nov 01 01:08:33.694599 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.694527 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced"} err="failed to get container status \"97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced\": rpc error: code = NotFound desc = could not find container \"97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced\": container with ID starting with 97ff1cc646e4524c75b703e887e912651c2d06a46a20677a180e40789bb15ced not found: ID does not exist" Nov 01 01:08:33.834099 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.834017 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222"] Nov 01 01:08:33.885255 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.885225 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-32439f-5589665d8d-kk222"] Nov 01 01:08:33.949615 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.949587 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8"] Nov 01 01:08:33.956769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:33.956743 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-32439f-5ddc866979-9xwn8"] Nov 01 01:08:34.147562 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.147541 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 01:08:34.194819 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.194791 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" path="/var/lib/kubelet/pods/0972708d-bace-41a7-a8f3-72d81578aeb2/volumes" Nov 01 01:08:34.195360 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.195344 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" path="/var/lib/kubelet/pods/86075eb8-bef0-4de5-b920-a7d36dff0f3c/volumes" Nov 01 01:08:34.225173 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225150 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-kube-api-access-66q8s\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-kube-api-access-66q8s\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225200 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-credential-socket\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225235 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55965aa-c817-4645-a465-6ba90a23486d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55965aa-c817-4645-a465-6ba90a23486d-istiod-ca-cert\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225287 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-data\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225314 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-socket\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225341 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-certs\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225383 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f55965aa-c817-4645-a465-6ba90a23486d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55965aa-c817-4645-a465-6ba90a23486d-istio-podinfo\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225428 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-istio-token\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-istio-token\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225519 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-data" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.225582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225535 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-socket" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.225582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225548 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-credential-socket" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.225582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225555 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-envoy\") pod \"f55965aa-c817-4645-a465-6ba90a23486d\" (UID: \"f55965aa-c817-4645-a465-6ba90a23486d\") " Nov 01 01:08:34.225913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.225695 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55965aa-c817-4645-a465-6ba90a23486d-istiod-ca-cert" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:34.226105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.226060 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-certs" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.226784 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.226760 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.226899 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.226788 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55965aa-c817-4645-a465-6ba90a23486d-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.226899 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.226802 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.226899 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.226818 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.226899 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.226833 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.227942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.227916 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-istio-token" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:34.228063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.228003 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f55965aa-c817-4645-a465-6ba90a23486d-istio-podinfo" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:08:34.228143 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.228123 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-kube-api-access-66q8s" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "kube-api-access-66q8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:34.228210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.228191 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-envoy" pod "f55965aa-c817-4645-a465-6ba90a23486d" (UID: "f55965aa-c817-4645-a465-6ba90a23486d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:08:34.328082 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.328052 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55965aa-c817-4645-a465-6ba90a23486d-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.328082 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.328082 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66q8s\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-kube-api-access-66q8s\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.328283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.328093 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55965aa-c817-4645-a465-6ba90a23486d-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.328283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.328101 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55965aa-c817-4645-a465-6ba90a23486d-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:34.681510 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.681479 4910 generic.go:356] "Generic (PLEG): container finished" podID="f55965aa-c817-4645-a465-6ba90a23486d" containerID="32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613" exitCode=0 Nov 01 01:08:34.682080 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.681538 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" Nov 01 01:08:34.682080 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.681558 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" event={"ID":"f55965aa-c817-4645-a465-6ba90a23486d","Type":"ContainerDied","Data":"32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613"} Nov 01 01:08:34.682080 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.681613 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx" event={"ID":"f55965aa-c817-4645-a465-6ba90a23486d","Type":"ContainerDied","Data":"b24c15a63f092e58c98ef81334ac9ce445744e9844af8b4650c6aa85449937c9"} Nov 01 01:08:34.682080 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.681649 4910 scope.go:122] "RemoveContainer" containerID="32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613" Nov 01 01:08:34.693317 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.693296 4910 scope.go:122] "RemoveContainer" containerID="32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613" Nov 01 01:08:34.693606 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:34.693585 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613\": container with ID starting with 32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613 not found: ID does not exist" containerID="32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613" Nov 01 01:08:34.693718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.693613 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613"} err="failed to get container status \"32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613\": rpc error: code = NotFound desc = could not find container \"32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613\": container with ID starting with 32cf79c38f5ee841979c7b4ab2bb2e446033a92a4e72ad66df003abc40c03613 not found: ID does not exist" Nov 01 01:08:34.849778 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.849747 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx"] Nov 01 01:08:34.899527 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:34.899490 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-948e22-openshift-default-5f6df6b99f-t79vx"] Nov 01 01:08:36.192743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:36.192713 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55965aa-c817-4645-a465-6ba90a23486d" path="/var/lib/kubelet/pods/f55965aa-c817-4645-a465-6ba90a23486d/volumes" Nov 01 01:08:43.103743 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:08:43.103708 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:08:49.627079 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627050 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd"] Nov 01 01:08:49.627601 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627588 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerName="payload-pre-processing" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627602 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerName="payload-pre-processing" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627614 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627636 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627646 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627651 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627663 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627667 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627683 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7cb2adde-1af6-4023-962c-1f5376d964f5" containerName="revoke-keys" Nov 01 01:08:49.627688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627687 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb2adde-1af6-4023-962c-1f5376d964f5" containerName="revoke-keys" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627694 4910 cpu_manager.go:414] "containerMap: removing container" podUID="457aa14d-cf8a-439f-8048-1f50d4431442" containerName="revoke-keys" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627699 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457aa14d-cf8a-439f-8048-1f50d4431442" containerName="revoke-keys" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627723 4910 cpu_manager.go:414] "containerMap: removing container" podUID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" containerName="payload-processing" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627727 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" containerName="payload-processing" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627806 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="86075eb8-bef0-4de5-b920-a7d36dff0f3c" containerName="payload-processing" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627818 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0972708d-bace-41a7-a8f3-72d81578aeb2" containerName="payload-pre-processing" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627824 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a66a46-af99-4275-9ed6-1c513dd1f327" containerName="istio-proxy" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627829 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b5e5da-310d-469f-b7b3-cd6f08314deb" containerName="main" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627836 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55965aa-c817-4645-a465-6ba90a23486d" containerName="istio-proxy" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627842 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb2adde-1af6-4023-962c-1f5376d964f5" containerName="revoke-keys" Nov 01 01:08:49.627972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.627848 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="457aa14d-cf8a-439f-8048-1f50d4431442" containerName="revoke-keys" Nov 01 01:08:49.639477 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.639452 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.692123 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.692089 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd"] Nov 01 01:08:49.736126 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.736094 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:08:49.736283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.736199 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:08:49.741862 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.741832 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckwdn\" (UniqueName: \"kubernetes.io/projected/8f12c3ea-c8e8-434a-8a44-deb3104eb199-kube-api-access-ckwdn\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.741996 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.741889 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.742073 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.742054 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.842720 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.842694 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckwdn\" (UniqueName: \"kubernetes.io/projected/8f12c3ea-c8e8-434a-8a44-deb3104eb199-kube-api-access-ckwdn\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.842898 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.842735 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.842898 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.842815 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.843374 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.843353 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.845304 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.845278 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.903769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.903700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckwdn\" (UniqueName: \"kubernetes.io/projected/8f12c3ea-c8e8-434a-8a44-deb3104eb199-kube-api-access-ckwdn\") pod \"maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:49.964682 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:49.964653 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:50.354356 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:50.354325 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd"] Nov 01 01:08:50.355779 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:08:50.355737 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f12c3ea_c8e8_434a_8a44_deb3104eb199.slice/crio-feab5bd85b8978ee8ee2634a6c0f3273891596e42bd9753808705a3ec50cd7cf WatchSource:0}: Error finding container feab5bd85b8978ee8ee2634a6c0f3273891596e42bd9753808705a3ec50cd7cf: Status 404 returned error can't find the container with id feab5bd85b8978ee8ee2634a6c0f3273891596e42bd9753808705a3ec50cd7cf Nov 01 01:08:50.841614 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:50.841569 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" event={"ID":"8f12c3ea-c8e8-434a-8a44-deb3104eb199","Type":"ContainerStarted","Data":"a723cb6b3e803a4d810a36bc8be73c3e993d3f031f7014d4d6aa5d2f48a4ebfe"} Nov 01 01:08:50.842108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:50.841647 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" event={"ID":"8f12c3ea-c8e8-434a-8a44-deb3104eb199","Type":"ContainerStarted","Data":"feab5bd85b8978ee8ee2634a6c0f3273891596e42bd9753808705a3ec50cd7cf"} Nov 01 01:08:50.957455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:50.957345 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" podStartSLOduration=1.9573277610000002 podStartE2EDuration="1.957327761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:08:50.956812014 +0000 UTC m=+2720.936394350" watchObservedRunningTime="2026-11-01 01:08:50.957327761 +0000 UTC m=+2720.936910095" Nov 01 01:08:51.850827 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:51.850792 4910 generic.go:356] "Generic (PLEG): container finished" podID="8f12c3ea-c8e8-434a-8a44-deb3104eb199" containerID="a723cb6b3e803a4d810a36bc8be73c3e993d3f031f7014d4d6aa5d2f48a4ebfe" exitCode=0 Nov 01 01:08:51.851258 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:51.850879 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" event={"ID":"8f12c3ea-c8e8-434a-8a44-deb3104eb199","Type":"ContainerDied","Data":"a723cb6b3e803a4d810a36bc8be73c3e993d3f031f7014d4d6aa5d2f48a4ebfe"} Nov 01 01:08:53.022490 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.022461 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:08:53.181384 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.181307 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-openshift-ca\") pod \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " Nov 01 01:08:53.181384 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.181376 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f12c3ea-c8e8-434a-8a44-deb3104eb199-kube-api-access-ckwdn\" (UniqueName: \"kubernetes.io/projected/8f12c3ea-c8e8-434a-8a44-deb3104eb199-kube-api-access-ckwdn\") pod \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " Nov 01 01:08:53.181660 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.181404 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-xks-ca\") pod \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\" (UID: \"8f12c3ea-c8e8-434a-8a44-deb3104eb199\") " Nov 01 01:08:53.181802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.181774 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-openshift-ca" pod "8f12c3ea-c8e8-434a-8a44-deb3104eb199" (UID: "8f12c3ea-c8e8-434a-8a44-deb3104eb199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:08:53.182029 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.182002 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:53.183772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.183748 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-xks-ca" pod "8f12c3ea-c8e8-434a-8a44-deb3104eb199" (UID: "8f12c3ea-c8e8-434a-8a44-deb3104eb199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:08:53.184230 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.184207 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f12c3ea-c8e8-434a-8a44-deb3104eb199-kube-api-access-ckwdn" pod "8f12c3ea-c8e8-434a-8a44-deb3104eb199" (UID: "8f12c3ea-c8e8-434a-8a44-deb3104eb199"). InnerVolumeSpecName "kube-api-access-ckwdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:08:53.282773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.282734 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f12c3ea-c8e8-434a-8a44-deb3104eb199-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:53.282773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.282773 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckwdn\" (UniqueName: \"kubernetes.io/projected/8f12c3ea-c8e8-434a-8a44-deb3104eb199-kube-api-access-ckwdn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:08:53.871142 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.871104 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" event={"ID":"8f12c3ea-c8e8-434a-8a44-deb3104eb199","Type":"ContainerDied","Data":"feab5bd85b8978ee8ee2634a6c0f3273891596e42bd9753808705a3ec50cd7cf"} Nov 01 01:08:53.871359 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.871191 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feab5bd85b8978ee8ee2634a6c0f3273891596e42bd9753808705a3ec50cd7cf" Nov 01 01:08:53.871432 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:08:53.871393 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd" Nov 01 01:09:01.624392 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:01.624353 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4"] Nov 01 01:09:01.628847 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:01.624704 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" containerID="cri-o://1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec" gracePeriod=30 Nov 01 01:09:01.629416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:01.629391 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc"] Nov 01 01:09:01.629606 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:01.629580 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" containerID="cri-o://62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e" gracePeriod=30 Nov 01 01:09:01.648066 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:01.648037 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9"] Nov 01 01:09:01.648269 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:01.648246 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" containerID="cri-o://83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead" gracePeriod=30 Nov 01 01:09:02.297897 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:02.297864 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:09:02.298156 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:02.297936 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:09:02.497315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:02.497278 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:09:02.497485 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:02.497353 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:09:03.642183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:03.642148 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5"] Nov 01 01:09:03.642710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:03.642413 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" containerID="cri-o://4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f" gracePeriod=30 Nov 01 01:09:06.256516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.256485 4910 patch_prober.go:29] interesting pod/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:09:06.257009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.256540 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:09:06.936661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.933950 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 01:09:06.964745 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.964720 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:09:06.970684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.970617 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-data\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.970824 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.970794 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-kube-api-access-ch2w7\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-kube-api-access-ch2w7\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.970879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.970823 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-certs\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.970879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.970829 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-data" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:06.970973 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.970879 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istiod-ca-cert\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.971036 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.970972 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-socket\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.971036 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971006 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-credential-socket\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.971153 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971061 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-token\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.971153 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971100 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-envoy\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.971153 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971141 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-podinfo\") pod \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\" (UID: \"c3e9c1df-cc91-4aed-8e1a-eb51042c031d\") " Nov 01 01:09:06.971322 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971159 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-certs" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:06.971322 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971249 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istiod-ca-cert" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:06.971446 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971319 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-credential-socket" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:06.971699 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971676 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.971833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971702 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.971833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971717 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.971833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.971731 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:06.972183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.972165 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-socket" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:06.973378 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.973357 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-envoy" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:06.973478 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.973463 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-podinfo" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:06.973805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.973784 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-kube-api-access-ch2w7" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "kube-api-access-ch2w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:06.973805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.973796 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-token" pod "c3e9c1df-cc91-4aed-8e1a-eb51042c031d" (UID: "c3e9c1df-cc91-4aed-8e1a-eb51042c031d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:06.975214 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.975195 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:09:06.988255 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.988231 4910 generic.go:356] "Generic (PLEG): container finished" podID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerID="1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec" exitCode=0 Nov 01 01:09:06.988347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.988286 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" Nov 01 01:09:06.988445 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.988411 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" event={"ID":"c3e9c1df-cc91-4aed-8e1a-eb51042c031d","Type":"ContainerDied","Data":"1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec"} Nov 01 01:09:06.988516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.988483 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4" event={"ID":"c3e9c1df-cc91-4aed-8e1a-eb51042c031d","Type":"ContainerDied","Data":"c5da9d938bf33be30e5f3c228c863ea055b72b021132a79ddae7192d8c97e03a"} Nov 01 01:09:06.988579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.988516 4910 scope.go:122] "RemoveContainer" containerID="1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec" Nov 01 01:09:06.990099 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.990076 4910 generic.go:356] "Generic (PLEG): container finished" podID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerID="62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e" exitCode=0 Nov 01 01:09:06.990372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.990143 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" Nov 01 01:09:06.990372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.990160 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" event={"ID":"0bf97496-8a64-4a79-8f7c-5135cae1279d","Type":"ContainerDied","Data":"62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e"} Nov 01 01:09:06.990372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.990216 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc" event={"ID":"0bf97496-8a64-4a79-8f7c-5135cae1279d","Type":"ContainerDied","Data":"cf6be70ed2df7d9a5f4907551c1b19eff5ab0afef48a75347e1f36e266e319d5"} Nov 01 01:09:06.991749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.991729 4910 generic.go:356] "Generic (PLEG): container finished" podID="5d0452ae-29cc-4144-985e-deb83634b42b" containerID="83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead" exitCode=0 Nov 01 01:09:06.991881 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.991762 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" Nov 01 01:09:06.991881 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.991763 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" event={"ID":"5d0452ae-29cc-4144-985e-deb83634b42b","Type":"ContainerDied","Data":"83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead"} Nov 01 01:09:06.991881 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:06.991859 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9" event={"ID":"5d0452ae-29cc-4144-985e-deb83634b42b","Type":"ContainerDied","Data":"2a72abc644ef1e69f1b3c5ecc7feabad34b6c58fa7a069eaf59be5f58de2e7d8"} Nov 01 01:09:07.000432 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.000416 4910 scope.go:122] "RemoveContainer" containerID="34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b" Nov 01 01:09:07.009726 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.009705 4910 scope.go:122] "RemoveContainer" containerID="1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec" Nov 01 01:09:07.010004 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:07.009986 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec\": container with ID starting with 1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec not found: ID does not exist" containerID="1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec" Nov 01 01:09:07.010083 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.010015 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec"} err="failed to get container status \"1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec\": rpc error: code = NotFound desc = could not find container \"1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec\": container with ID starting with 1b07242f463b355baffdc44f619e75d9076c80a05a194f5125f86836ce4a95ec not found: ID does not exist" Nov 01 01:09:07.010083 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.010038 4910 scope.go:122] "RemoveContainer" containerID="34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b" Nov 01 01:09:07.010279 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:07.010262 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b\": container with ID starting with 34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b not found: ID does not exist" containerID="34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b" Nov 01 01:09:07.010320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.010284 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b"} err="failed to get container status \"34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b\": rpc error: code = NotFound desc = could not find container \"34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b\": container with ID starting with 34844ccff1c71cdac84a380b9de5855b14d41618e895ec71939c78fcfd7e799b not found: ID does not exist" Nov 01 01:09:07.010320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.010299 4910 scope.go:122] "RemoveContainer" containerID="62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e" Nov 01 01:09:07.019564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.019546 4910 scope.go:122] "RemoveContainer" containerID="62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e" Nov 01 01:09:07.019846 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:07.019828 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e\": container with ID starting with 62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e not found: ID does not exist" containerID="62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e" Nov 01 01:09:07.019914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.019851 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e"} err="failed to get container status \"62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e\": rpc error: code = NotFound desc = could not find container \"62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e\": container with ID starting with 62c694c158a9710919369fa49723703408f59d6fdd557b3b8becaa4edd0f6d4e not found: ID does not exist" Nov 01 01:09:07.019914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.019868 4910 scope.go:122] "RemoveContainer" containerID="83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead" Nov 01 01:09:07.028994 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.028979 4910 scope.go:122] "RemoveContainer" containerID="83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead" Nov 01 01:09:07.029265 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:07.029244 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead\": container with ID starting with 83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead not found: ID does not exist" containerID="83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead" Nov 01 01:09:07.029333 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.029271 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead"} err="failed to get container status \"83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead\": rpc error: code = NotFound desc = could not find container \"83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead\": container with ID starting with 83f201fb2f0377a12e2a74a9afab9f06c73612b62c8b5cd3b55d575712213ead not found: ID does not exist" Nov 01 01:09:07.072673 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072646 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-kube-api-access-hz68g\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-kube-api-access-hz68g\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.072802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-credential-socket\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.072802 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072777 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-envoy\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.072954 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072818 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-socket\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.072954 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072874 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-istio-token\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-istio-token\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.072954 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072912 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d0452ae-29cc-4144-985e-deb83634b42b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0452ae-29cc-4144-985e-deb83634b42b-istio-podinfo\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.072954 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072945 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-certs\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.073193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072990 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-envoy\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.073193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.072995 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-credential-socket" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.073193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073022 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-certs\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.073193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073056 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-data\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.073193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073094 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-kube-api-access-gw2db\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-kube-api-access-gw2db\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073199 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-token\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-token\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073217 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-certs" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073221 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-socket" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073237 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0452ae-29cc-4144-985e-deb83634b42b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0452ae-29cc-4144-985e-deb83634b42b-istiod-ca-cert\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073292 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-credential-socket\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073344 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-data" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073349 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-data\") pod \"5d0452ae-29cc-4144-985e-deb83634b42b\" (UID: \"5d0452ae-29cc-4144-985e-deb83634b42b\") " Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073356 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-certs" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.073463 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073452 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf97496-8a64-4a79-8f7c-5135cae1279d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf97496-8a64-4a79-8f7c-5135cae1279d-istiod-ca-cert\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.073989 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073603 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-socket\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.073989 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073608 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-data" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.073989 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073661 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-podinfo\") pod \"0bf97496-8a64-4a79-8f7c-5135cae1279d\" (UID: \"0bf97496-8a64-4a79-8f7c-5135cae1279d\") " Nov 01 01:09:07.073989 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.073953 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0452ae-29cc-4144-985e-deb83634b42b-istiod-ca-cert" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:07.074216 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074185 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-credential-socket" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.074384 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074345 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074384 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074365 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch2w7\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-kube-api-access-ch2w7\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074384 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074381 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074395 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074409 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074423 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0452ae-29cc-4144-985e-deb83634b42b-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074437 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074451 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074466 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074485 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074498 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074512 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e9c1df-cc91-4aed-8e1a-eb51042c031d-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.074557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074526 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.075171 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.074845 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf97496-8a64-4a79-8f7c-5135cae1279d-istiod-ca-cert" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:07.075961 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.075934 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-socket" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.076148 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.076125 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-envoy" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.077780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.077760 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d0452ae-29cc-4144-985e-deb83634b42b-istio-podinfo" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:07.079564 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.079525 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-istio-token" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.079675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.079578 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-kube-api-access-hz68g" pod "5d0452ae-29cc-4144-985e-deb83634b42b" (UID: "5d0452ae-29cc-4144-985e-deb83634b42b"). InnerVolumeSpecName "kube-api-access-hz68g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.079675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.079604 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-kube-api-access-gw2db" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "kube-api-access-gw2db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.079774 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.079698 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-envoy" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:07.079870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.079820 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-podinfo" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:07.079978 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.079961 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-token" pod "0bf97496-8a64-4a79-8f7c-5135cae1279d" (UID: "0bf97496-8a64-4a79-8f7c-5135cae1279d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:07.175991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.175918 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.175991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.175944 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0452ae-29cc-4144-985e-deb83634b42b-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.175991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.175957 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0452ae-29cc-4144-985e-deb83634b42b-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.175991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.175966 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2db\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-kube-api-access-gw2db\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.175991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.175976 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.175991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.175985 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bf97496-8a64-4a79-8f7c-5135cae1279d-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.175991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.175996 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.176327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.176004 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.176327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.176013 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz68g\" (UniqueName: \"kubernetes.io/projected/5d0452ae-29cc-4144-985e-deb83634b42b-kube-api-access-hz68g\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.176327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.176021 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bf97496-8a64-4a79-8f7c-5135cae1279d-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:07.347253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.347223 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4"] Nov 01 01:09:07.352062 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.352041 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-m2cl4"] Nov 01 01:09:07.583115 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.583083 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9"] Nov 01 01:09:07.588725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.588693 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-bt8g9"] Nov 01 01:09:07.698940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.698909 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc"] Nov 01 01:09:07.704832 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:07.704801 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b112c158-openshift-default-86bb44f7df-cgrbc"] Nov 01 01:09:08.226647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.223418 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" path="/var/lib/kubelet/pods/0bf97496-8a64-4a79-8f7c-5135cae1279d/volumes" Nov 01 01:09:08.226647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.224192 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" path="/var/lib/kubelet/pods/5d0452ae-29cc-4144-985e-deb83634b42b/volumes" Nov 01 01:09:08.226647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.224930 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" path="/var/lib/kubelet/pods/c3e9c1df-cc91-4aed-8e1a-eb51042c031d/volumes" Nov 01 01:09:08.759581 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.759541 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cfbd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 01 01:09:08.760154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.759653 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 01 01:09:08.948124 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.948097 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 01:09:08.997427 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997399 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-data\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997431 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-credential-socket\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997467 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istiod-ca-cert\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997496 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-kube-api-access-fqt6l\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-kube-api-access-fqt6l\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997516 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-envoy\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997536 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-socket\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997606 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-certs\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997691 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-token\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-token\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.997914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997738 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-credential-socket" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:08.997914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997758 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-podinfo\") pod \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\" (UID: \"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a\") " Nov 01 01:09:08.998150 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997755 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-data" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:08.998150 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.997946 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-certs" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:08.998150 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.998046 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istiod-ca-cert" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:08.998312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.998224 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.998312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.998246 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.998312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.998260 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.998312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.998275 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:08.998515 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:08.998404 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-socket" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:09.000224 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.000194 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-token" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:09.000364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.000277 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-podinfo" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:09.000447 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.000361 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-envoy" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:09.000590 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.000569 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-kube-api-access-fqt6l" pod "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" (UID: "9a2dd317-384d-4cbe-b4f3-2cd8cf04858a"). InnerVolumeSpecName "kube-api-access-fqt6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:09.061327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.061290 4910 generic.go:356] "Generic (PLEG): container finished" podID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerID="4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f" exitCode=0 Nov 01 01:09:09.061502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.061337 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" Nov 01 01:09:09.061502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.061369 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" event={"ID":"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a","Type":"ContainerDied","Data":"4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f"} Nov 01 01:09:09.061502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.061429 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5" event={"ID":"9a2dd317-384d-4cbe-b4f3-2cd8cf04858a","Type":"ContainerDied","Data":"8a015a261e003cc010d36ee86fc3840c6229cae8d51c7fcdf58b93ff7d153e51"} Nov 01 01:09:09.061502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.061458 4910 scope.go:122] "RemoveContainer" containerID="4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f" Nov 01 01:09:09.086018 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.085977 4910 scope.go:122] "RemoveContainer" containerID="4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f" Nov 01 01:09:09.086427 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:09.086399 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f\": container with ID starting with 4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f not found: ID does not exist" containerID="4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f" Nov 01 01:09:09.086522 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.086456 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f"} err="failed to get container status \"4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f\": rpc error: code = NotFound desc = could not find container \"4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f\": container with ID starting with 4beee7998ec007eae925c06cdfc6ecebc4bd52381d874573f6dcd2fd0118086f not found: ID does not exist" Nov 01 01:09:09.099059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.099033 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqt6l\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-kube-api-access-fqt6l\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:09.099147 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.099062 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:09.099147 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.099077 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:09.099147 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.099089 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:09.099147 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.099102 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:09.232895 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.232813 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5"] Nov 01 01:09:09.242562 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.242532 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cfbd5"] Nov 01 01:09:09.967211 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.967173 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg"] Nov 01 01:09:09.970519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:09.970491 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-32439f-xk9jg"] Nov 01 01:09:10.192954 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:10.192926 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284d5886-0fc2-4c05-99e8-11b5319419be" path="/var/lib/kubelet/pods/284d5886-0fc2-4c05-99e8-11b5319419be/volumes" Nov 01 01:09:10.193371 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:10.193359 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" path="/var/lib/kubelet/pods/9a2dd317-384d-4cbe-b4f3-2cd8cf04858a/volumes" Nov 01 01:09:13.105108 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:13.105073 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:09:15.596403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596364 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf"] Nov 01 01:09:15.596958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596876 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8f12c3ea-c8e8-434a-8a44-deb3104eb199" containerName="revoke-keys" Nov 01 01:09:15.596958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596887 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f12c3ea-c8e8-434a-8a44-deb3104eb199" containerName="revoke-keys" Nov 01 01:09:15.596958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596901 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" Nov 01 01:09:15.596958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596907 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" Nov 01 01:09:15.596958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596930 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" Nov 01 01:09:15.596958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596937 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" Nov 01 01:09:15.596958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596963 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596968 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596976 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596981 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596990 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.596994 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.597067 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.597075 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2dd317-384d-4cbe-b4f3-2cd8cf04858a" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.597081 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e9c1df-cc91-4aed-8e1a-eb51042c031d" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.597090 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f12c3ea-c8e8-434a-8a44-deb3104eb199" containerName="revoke-keys" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.597095 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf97496-8a64-4a79-8f7c-5135cae1279d" containerName="istio-proxy" Nov 01 01:09:15.597331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.597102 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0452ae-29cc-4144-985e-deb83634b42b" containerName="istio-proxy" Nov 01 01:09:15.602061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.602041 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.683693 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.683654 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.683855 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.683709 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74gj\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-kube-api-access-d74gj\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.683855 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.683763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.683990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.683870 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.683990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.683905 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.683990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.683929 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.684152 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.684002 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.684152 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.684123 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.684268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.684159 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.699097 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.699063 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf"] Nov 01 01:09:15.785026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.784999 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785037 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785063 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785111 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785165 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785240 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785264 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74gj\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-kube-api-access-d74gj\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785319 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785406 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785510 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-credential-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.785784 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785767 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-certs\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.786024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.785899 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-data\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.786173 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.786154 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-socket\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.786268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.786246 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istiod-ca-cert\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.787767 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.787732 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-envoy\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.788593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.788570 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-podinfo\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.846210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.846173 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-token\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.846363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.846309 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74gj\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-kube-api-access-d74gj\") pod \"e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:15.916526 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:15.916451 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:16.364115 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:16.364086 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf"] Nov 01 01:09:16.364757 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:09:16.364729 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe10bc93_94f5_4c8d_80b4_2c96b1b1cda4.slice/crio-93a7dd41a5397076b7dc74ee400883efd680ea9d5eeb986629d0adb220fb958a WatchSource:0}: Error finding container 93a7dd41a5397076b7dc74ee400883efd680ea9d5eeb986629d0adb220fb958a: Status 404 returned error can't find the container with id 93a7dd41a5397076b7dc74ee400883efd680ea9d5eeb986629d0adb220fb958a Nov 01 01:09:16.367533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:16.367503 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:16.367595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:16.367563 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:16.367675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:16.367596 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:17.204689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.204286 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5"] Nov 01 01:09:17.204689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.204529 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" containerID="cri-o://eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185" gracePeriod=30 Nov 01 01:09:17.209437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.207897 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf"] Nov 01 01:09:17.210550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.210520 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" event={"ID":"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4","Type":"ContainerStarted","Data":"06551ef93ad3bcf601ec0bc7c658772da64da1158916c2100e249bdc31fead5b"} Nov 01 01:09:17.210674 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.210565 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" event={"ID":"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4","Type":"ContainerStarted","Data":"93a7dd41a5397076b7dc74ee400883efd680ea9d5eeb986629d0adb220fb958a"} Nov 01 01:09:17.210750 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.210736 4910 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" secret="" err="secret \"e2e-worker-w0-32439f-openshift-default-dockercfg-n9rr6\" not found" Nov 01 01:09:17.485334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.485194 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" podStartSLOduration=2.48517368 podStartE2EDuration="2.48517368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:17.484838545 +0000 UTC m=+2747.464420915" watchObservedRunningTime="2026-11-01 01:09:17.48517368 +0000 UTC m=+2747.464756015" Nov 01 01:09:17.917132 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.917095 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:17.918812 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.918783 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 01 01:09:17.918949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:17.918839 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" podUID="fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 01 01:09:18.217173 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:18.217077 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" podUID="fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" containerName="istio-proxy" containerID="cri-o://06551ef93ad3bcf601ec0bc7c658772da64da1158916c2100e249bdc31fead5b" gracePeriod=30 Nov 01 01:09:22.764780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.764360 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 01:09:22.827611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-podinfo\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827588 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-credential-socket\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827812 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827658 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-kube-api-access-4z5jx\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-kube-api-access-4z5jx\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827812 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827717 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-socket\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827812 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827755 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-token\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-token\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827812 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827797 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62a8809-b3d0-4475-ac5d-b58536c47600-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62a8809-b3d0-4475-ac5d-b58536c47600-istiod-ca-cert\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827823 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-certs\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827883 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-data\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.827946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.827919 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-envoy\") pod \"a62a8809-b3d0-4475-ac5d-b58536c47600\" (UID: \"a62a8809-b3d0-4475-ac5d-b58536c47600\") " Nov 01 01:09:22.828552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.828276 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-socket" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:22.828552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.828400 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62a8809-b3d0-4475-ac5d-b58536c47600-istiod-ca-cert" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:22.829387 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.828833 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-credential-socket" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:22.829455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.829405 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-data" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:22.829693 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.829669 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-certs" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:22.830707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.830651 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-podinfo" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:22.831085 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.831063 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-envoy" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:22.831294 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.831265 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-kube-api-access-4z5jx" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "kube-api-access-4z5jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:22.831294 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.831277 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-token" pod "a62a8809-b3d0-4475-ac5d-b58536c47600" (UID: "a62a8809-b3d0-4475-ac5d-b58536c47600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:22.929490 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929454 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929490 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929484 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929498 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z5jx\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-kube-api-access-4z5jx\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929514 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929525 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929536 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62a8809-b3d0-4475-ac5d-b58536c47600-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929548 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929560 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:22.929727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:22.929572 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62a8809-b3d0-4475-ac5d-b58536c47600-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:23.327809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.327763 4910 generic.go:356] "Generic (PLEG): container finished" podID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerID="eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185" exitCode=0 Nov 01 01:09:23.328025 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.327821 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" event={"ID":"a62a8809-b3d0-4475-ac5d-b58536c47600","Type":"ContainerDied","Data":"eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185"} Nov 01 01:09:23.328025 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.327942 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" event={"ID":"a62a8809-b3d0-4475-ac5d-b58536c47600","Type":"ContainerDied","Data":"fae3f2f7e9e872df6e4e275f8426ce81c4c6f607da0bbc0689a000fa2ae4cd08"} Nov 01 01:09:23.328025 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.327993 4910 scope.go:122] "RemoveContainer" containerID="eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185" Nov 01 01:09:23.328200 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.328184 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5" Nov 01 01:09:23.344996 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.344848 4910 scope.go:122] "RemoveContainer" containerID="eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185" Nov 01 01:09:23.345314 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:23.345291 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185\": container with ID starting with eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185 not found: ID does not exist" containerID="eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185" Nov 01 01:09:23.345409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.345327 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185"} err="failed to get container status \"eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185\": rpc error: code = NotFound desc = could not find container \"eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185\": container with ID starting with eccb1f604d0909f7d966b477c13b67547f5b64aba7671750786b1da56e034185 not found: ID does not exist" Nov 01 01:09:23.535771 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.535738 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5"] Nov 01 01:09:23.540262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:23.540237 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-k5mn5"] Nov 01 01:09:24.192952 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:24.192907 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" path="/var/lib/kubelet/pods/a62a8809-b3d0-4475-ac5d-b58536c47600/volumes" Nov 01 01:09:29.374795 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.374756 4910 generic.go:356] "Generic (PLEG): container finished" podID="fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" containerID="06551ef93ad3bcf601ec0bc7c658772da64da1158916c2100e249bdc31fead5b" exitCode=0 Nov 01 01:09:29.375176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.374836 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" event={"ID":"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4","Type":"ContainerDied","Data":"06551ef93ad3bcf601ec0bc7c658772da64da1158916c2100e249bdc31fead5b"} Nov 01 01:09:29.483720 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.483699 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:29.500434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500397 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-podinfo\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500454 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-certs\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500476 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istiod-ca-cert\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500533 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-kube-api-access-d74gj\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-kube-api-access-d74gj\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500555 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-envoy\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500586 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-data\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500907 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500654 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-socket\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500907 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500687 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-credential-socket\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.500907 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500780 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-token\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-token\") pod \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\" (UID: \"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4\") " Nov 01 01:09:29.501112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500933 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-data" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:29.501112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.500947 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-socket" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:29.501112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501001 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-certs" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:29.501266 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501159 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-credential-socket" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:29.501266 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501226 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istiod-ca-cert" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:29.501485 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501465 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.501551 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501493 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.501551 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501509 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.501551 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501525 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.501551 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.501540 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.503123 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.503094 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-envoy" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:09:29.503123 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.503097 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-token" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:29.503335 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.503321 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-podinfo" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:09:29.503566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.503546 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-kube-api-access-d74gj" pod "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" (UID: "fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4"). InnerVolumeSpecName "kube-api-access-d74gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:29.604550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.603285 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74gj\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-kube-api-access-d74gj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.604550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.604454 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.604550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.604508 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:29.604550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:29.604525 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:30.383069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:30.383029 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" Nov 01 01:09:30.383688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:30.383034 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf" event={"ID":"fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4","Type":"ContainerDied","Data":"93a7dd41a5397076b7dc74ee400883efd680ea9d5eeb986629d0adb220fb958a"} Nov 01 01:09:30.383688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:30.383166 4910 scope.go:122] "RemoveContainer" containerID="06551ef93ad3bcf601ec0bc7c658772da64da1158916c2100e249bdc31fead5b" Nov 01 01:09:30.550455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:30.550407 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf"] Nov 01 01:09:30.600444 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:30.600407 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-32439f-openshift-default-74c574d76c-4spzf"] Nov 01 01:09:32.191929 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:32.191896 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" path="/var/lib/kubelet/pods/fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4/volumes" Nov 01 01:09:32.744931 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:32.744902 4910 scope.go:122] "RemoveContainer" containerID="2a9bd4a527741de5f380307d848f8b057b4f8c4e3585c4d9d00467a5bddd4711" Nov 01 01:09:32.755295 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:32.755276 4910 scope.go:122] "RemoveContainer" containerID="f4ee5d88f9b7e1731ebbbb6bbfdd767cc45f0187ef0d153e31a5e37ee6abf764" Nov 01 01:09:32.764809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:32.764791 4910 scope.go:122] "RemoveContainer" containerID="1c083ea601f09fedc3581045407750ecf73c99b07efe205572310e8611b87153" Nov 01 01:09:33.604520 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.604485 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7"] Nov 01 01:09:33.605028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.605013 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" containerName="istio-proxy" Nov 01 01:09:33.605074 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.605029 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" containerName="istio-proxy" Nov 01 01:09:33.605074 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.605061 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" Nov 01 01:09:33.605074 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.605066 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" Nov 01 01:09:33.605176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.605151 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62a8809-b3d0-4475-ac5d-b58536c47600" containerName="istio-proxy" Nov 01 01:09:33.605176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.605163 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe10bc93-94f5-4c8d-80b4-2c96b1b1cda4" containerName="istio-proxy" Nov 01 01:09:33.610554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.610534 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.648797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.648763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.649007 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.648825 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7q9z\" (UniqueName: \"kubernetes.io/projected/04426f20-1fc6-453e-8bad-72097fed7296-kube-api-access-s7q9z\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.649007 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.648914 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.664312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.664242 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:09:33.668275 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.668238 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7"] Nov 01 01:09:33.713838 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.713794 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:09:33.750333 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.750301 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.750556 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.750356 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7q9z\" (UniqueName: \"kubernetes.io/projected/04426f20-1fc6-453e-8bad-72097fed7296-kube-api-access-s7q9z\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.750556 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.750480 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.751130 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.751102 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.752942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.752914 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.812403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.812372 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7q9z\" (UniqueName: \"kubernetes.io/projected/04426f20-1fc6-453e-8bad-72097fed7296-kube-api-access-s7q9z\") pod \"maas-api-revoke-keys-e2e-api-a-718e19-wbzl7\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:33.924510 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:33.924430 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:34.305693 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:09:34.305664 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04426f20_1fc6_453e_8bad_72097fed7296.slice/crio-ff32522cf29a9d4276380619b5c9f8006375f10ccba0c41f6a48e857fecb2a72 WatchSource:0}: Error finding container ff32522cf29a9d4276380619b5c9f8006375f10ccba0c41f6a48e857fecb2a72: Status 404 returned error can't find the container with id ff32522cf29a9d4276380619b5c9f8006375f10ccba0c41f6a48e857fecb2a72 Nov 01 01:09:34.305808 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:34.305756 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7"] Nov 01 01:09:34.414184 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:34.414145 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" event={"ID":"04426f20-1fc6-453e-8bad-72097fed7296","Type":"ContainerStarted","Data":"ff32522cf29a9d4276380619b5c9f8006375f10ccba0c41f6a48e857fecb2a72"} Nov 01 01:09:35.421818 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:35.421724 4910 generic.go:356] "Generic (PLEG): container finished" podID="04426f20-1fc6-453e-8bad-72097fed7296" containerID="5abebda7e9ef87f83c60634f0564e56e2f07b3ec2c0d646e595b3555ecdfb082" exitCode=0 Nov 01 01:09:35.422271 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:35.421807 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" event={"ID":"04426f20-1fc6-453e-8bad-72097fed7296","Type":"ContainerDied","Data":"5abebda7e9ef87f83c60634f0564e56e2f07b3ec2c0d646e595b3555ecdfb082"} Nov 01 01:09:36.601879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.601852 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:36.681866 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.681786 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-xks-ca\") pod \"04426f20-1fc6-453e-8bad-72097fed7296\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " Nov 01 01:09:36.682027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.681916 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-openshift-ca\") pod \"04426f20-1fc6-453e-8bad-72097fed7296\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " Nov 01 01:09:36.682027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.681960 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04426f20-1fc6-453e-8bad-72097fed7296-kube-api-access-s7q9z\" (UniqueName: \"kubernetes.io/projected/04426f20-1fc6-453e-8bad-72097fed7296-kube-api-access-s7q9z\") pod \"04426f20-1fc6-453e-8bad-72097fed7296\" (UID: \"04426f20-1fc6-453e-8bad-72097fed7296\") " Nov 01 01:09:36.682330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.682296 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-openshift-ca" pod "04426f20-1fc6-453e-8bad-72097fed7296" (UID: "04426f20-1fc6-453e-8bad-72097fed7296"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:09:36.684198 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.684170 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-xks-ca" pod "04426f20-1fc6-453e-8bad-72097fed7296" (UID: "04426f20-1fc6-453e-8bad-72097fed7296"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:09:36.684427 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.684406 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04426f20-1fc6-453e-8bad-72097fed7296-kube-api-access-s7q9z" pod "04426f20-1fc6-453e-8bad-72097fed7296" (UID: "04426f20-1fc6-453e-8bad-72097fed7296"). InnerVolumeSpecName "kube-api-access-s7q9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:09:36.783127 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.783095 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:36.783281 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.783123 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04426f20-1fc6-453e-8bad-72097fed7296-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:36.783281 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:36.783148 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7q9z\" (UniqueName: \"kubernetes.io/projected/04426f20-1fc6-453e-8bad-72097fed7296-kube-api-access-s7q9z\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:09:37.438900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:37.438859 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" event={"ID":"04426f20-1fc6-453e-8bad-72097fed7296","Type":"ContainerDied","Data":"ff32522cf29a9d4276380619b5c9f8006375f10ccba0c41f6a48e857fecb2a72"} Nov 01 01:09:37.438900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:37.438884 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7" Nov 01 01:09:37.439117 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:37.438921 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff32522cf29a9d4276380619b5c9f8006375f10ccba0c41f6a48e857fecb2a72" Nov 01 01:09:43.102636 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:09:43.102583 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:09:49.112679 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.112649 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft"] Nov 01 01:09:49.113178 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.113107 4910 cpu_manager.go:414] "containerMap: removing container" podUID="04426f20-1fc6-453e-8bad-72097fed7296" containerName="revoke-keys" Nov 01 01:09:49.113178 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.113118 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04426f20-1fc6-453e-8bad-72097fed7296" containerName="revoke-keys" Nov 01 01:09:49.113253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.113220 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="04426f20-1fc6-453e-8bad-72097fed7296" containerName="revoke-keys" Nov 01 01:09:49.116496 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.116480 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.216408 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216350 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.216648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216412 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.216648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216441 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.216648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216504 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.216648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216544 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40aac9-e330-4270-8d4e-5770a83b105e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.216648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216582 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.216648 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216611 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5l62\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-kube-api-access-h5l62\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.217046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216781 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.217046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.216858 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.233767 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.233733 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft"] Nov 01 01:09:49.317470 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317427 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.317470 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317476 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.317762 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317493 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.317762 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.317762 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317558 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40aac9-e330-4270-8d4e-5770a83b105e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.317762 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317591 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318006 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317828 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5l62\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-kube-api-access-h5l62\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318006 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317890 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318006 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317961 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318006 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.317979 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318218 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.318013 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318218 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.318163 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318346 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.318316 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.318471 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.318451 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40aac9-e330-4270-8d4e-5770a83b105e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.320252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.320228 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.320352 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.320326 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.379321 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.379259 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5l62\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-kube-api-access-h5l62\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.379321 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.379275 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pqmft\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.429465 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.429435 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:49.860892 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.860860 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft"] Nov 01 01:09:49.861986 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:09:49.861962 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d40aac9_e330_4270_8d4e_5770a83b105e.slice/crio-94c7282e7a5bd3b3aafdaf946492368847640ef75bdf88201c0282bde6865d3b WatchSource:0}: Error finding container 94c7282e7a5bd3b3aafdaf946492368847640ef75bdf88201c0282bde6865d3b: Status 404 returned error can't find the container with id 94c7282e7a5bd3b3aafdaf946492368847640ef75bdf88201c0282bde6865d3b Nov 01 01:09:49.865192 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.865153 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:49.865284 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.865236 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:49.865284 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:49.865275 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:09:50.604573 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:50.604525 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" event={"ID":"5d40aac9-e330-4270-8d4e-5770a83b105e","Type":"ContainerStarted","Data":"5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd"} Nov 01 01:09:50.605130 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:50.604592 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" event={"ID":"5d40aac9-e330-4270-8d4e-5770a83b105e","Type":"ContainerStarted","Data":"94c7282e7a5bd3b3aafdaf946492368847640ef75bdf88201c0282bde6865d3b"} Nov 01 01:09:50.731507 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:50.731431 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podStartSLOduration=1.731411354 podStartE2EDuration="1.731411354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:09:50.731214061 +0000 UTC m=+2780.710796413" watchObservedRunningTime="2026-11-01 01:09:50.731411354 +0000 UTC m=+2780.710993690" Nov 01 01:09:51.430542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:51.430513 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:51.432421 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:51.432387 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pqmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 01 01:09:51.432553 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:51.432434 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 01 01:09:52.429864 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:52.429828 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pqmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 01 01:09:52.430284 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:52.429881 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 01 01:09:53.430582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:53.430542 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pqmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 01 01:09:53.431022 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:53.430615 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 01 01:09:54.430106 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:54.430073 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pqmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 01 01:09:54.430303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:54.430131 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 01 01:09:55.430181 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:55.430146 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pqmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 01 01:09:55.430679 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:55.430211 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 01 01:09:56.430240 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:56.430200 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pqmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 01 01:09:56.430724 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:56.430263 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 01 01:09:57.429949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:57.429913 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pqmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 01 01:09:57.430134 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:57.429973 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 01 01:09:58.434275 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:58.434246 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:58.434749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:58.434559 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:09:58.435360 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:09:58.435341 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:10:13.104220 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:10:13.104178 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:10:17.653387 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:17.653355 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd"] Nov 01 01:10:17.702975 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:17.702943 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-603e8c-jv2qd"] Nov 01 01:10:18.136667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.136602 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n"] Nov 01 01:10:18.140719 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.140696 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.193036 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.193009 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f12c3ea-c8e8-434a-8a44-deb3104eb199" path="/var/lib/kubelet/pods/8f12c3ea-c8e8-434a-8a44-deb3104eb199/volumes" Nov 01 01:10:18.199670 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.199643 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n"] Nov 01 01:10:18.245339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.245315 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:10:18.245339 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.245324 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:10:18.252889 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.252868 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.253013 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.252930 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.253059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.253043 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6jk\" (UniqueName: \"kubernetes.io/projected/2d8669ca-2949-4edd-8706-815cfe7055a0-kube-api-access-zr6jk\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.354152 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.354118 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.354364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.354176 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6jk\" (UniqueName: \"kubernetes.io/projected/2d8669ca-2949-4edd-8706-815cfe7055a0-kube-api-access-zr6jk\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.354364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.354262 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.354818 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.354795 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.356946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.356921 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.418362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.418283 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6jk\" (UniqueName: \"kubernetes.io/projected/2d8669ca-2949-4edd-8706-815cfe7055a0-kube-api-access-zr6jk\") pod \"maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.455369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.455343 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:18.833664 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:10:18.833609 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8669ca_2949_4edd_8706_815cfe7055a0.slice/crio-18001d99ef90cfd6c5d0c5df4e4d9ef91e2e4f7c5c5b53b126d0de25a9716dea WatchSource:0}: Error finding container 18001d99ef90cfd6c5d0c5df4e4d9ef91e2e4f7c5c5b53b126d0de25a9716dea: Status 404 returned error can't find the container with id 18001d99ef90cfd6c5d0c5df4e4d9ef91e2e4f7c5c5b53b126d0de25a9716dea Nov 01 01:10:18.834028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.833711 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n"] Nov 01 01:10:18.853613 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:18.853580 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" event={"ID":"2d8669ca-2949-4edd-8706-815cfe7055a0","Type":"ContainerStarted","Data":"18001d99ef90cfd6c5d0c5df4e4d9ef91e2e4f7c5c5b53b126d0de25a9716dea"} Nov 01 01:10:19.866617 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:19.866580 4910 generic.go:356] "Generic (PLEG): container finished" podID="2d8669ca-2949-4edd-8706-815cfe7055a0" containerID="e69c4b9c2f3520358232a3a03aba78b0294619541707313d861aba596eccdfb9" exitCode=0 Nov 01 01:10:19.867086 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:19.866662 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" event={"ID":"2d8669ca-2949-4edd-8706-815cfe7055a0","Type":"ContainerDied","Data":"e69c4b9c2f3520358232a3a03aba78b0294619541707313d861aba596eccdfb9"} Nov 01 01:10:20.822167 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.822134 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs"] Nov 01 01:10:20.826003 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.825987 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.884143 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.884108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.884481 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.884294 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2j49\" (UniqueName: \"kubernetes.io/projected/339fac2d-201a-4c40-b722-366e74518ce4-kube-api-access-j2j49\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.884481 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.884416 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.899676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.899641 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs"] Nov 01 01:10:20.985375 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.985320 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2j49\" (UniqueName: \"kubernetes.io/projected/339fac2d-201a-4c40-b722-366e74518ce4-kube-api-access-j2j49\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.985577 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.985425 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.985577 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.985522 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.986341 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.986310 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:20.988487 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:20.988461 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:21.048408 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.048384 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2j49\" (UniqueName: \"kubernetes.io/projected/339fac2d-201a-4c40-b722-366e74518ce4-kube-api-access-j2j49\") pod \"maas-api-revoke-sub-f46ec952-m6bqs\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:21.048534 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.048416 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:21.086519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.086463 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-openshift-ca\") pod \"2d8669ca-2949-4edd-8706-815cfe7055a0\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " Nov 01 01:10:21.086649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.086516 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-xks-ca\") pod \"2d8669ca-2949-4edd-8706-815cfe7055a0\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " Nov 01 01:10:21.086649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.086594 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8669ca-2949-4edd-8706-815cfe7055a0-kube-api-access-zr6jk\" (UniqueName: \"kubernetes.io/projected/2d8669ca-2949-4edd-8706-815cfe7055a0-kube-api-access-zr6jk\") pod \"2d8669ca-2949-4edd-8706-815cfe7055a0\" (UID: \"2d8669ca-2949-4edd-8706-815cfe7055a0\") " Nov 01 01:10:21.086821 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.086800 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-openshift-ca" pod "2d8669ca-2949-4edd-8706-815cfe7055a0" (UID: "2d8669ca-2949-4edd-8706-815cfe7055a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:10:21.086979 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.086961 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:21.088638 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.088608 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-xks-ca" pod "2d8669ca-2949-4edd-8706-815cfe7055a0" (UID: "2d8669ca-2949-4edd-8706-815cfe7055a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:10:21.088833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.088813 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8669ca-2949-4edd-8706-815cfe7055a0-kube-api-access-zr6jk" pod "2d8669ca-2949-4edd-8706-815cfe7055a0" (UID: "2d8669ca-2949-4edd-8706-815cfe7055a0"). InnerVolumeSpecName "kube-api-access-zr6jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:21.140064 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.140038 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:21.187943 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.187904 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6jk\" (UniqueName: \"kubernetes.io/projected/2d8669ca-2949-4edd-8706-815cfe7055a0-kube-api-access-zr6jk\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:21.187943 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.187946 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8669ca-2949-4edd-8706-815cfe7055a0-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:21.520191 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.520166 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs"] Nov 01 01:10:21.521109 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:10:21.521083 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339fac2d_201a_4c40_b722_366e74518ce4.slice/crio-42b983056e413fcecf4a870c0e625f3c416431562cb77efaecdb154b372c0f17 WatchSource:0}: Error finding container 42b983056e413fcecf4a870c0e625f3c416431562cb77efaecdb154b372c0f17: Status 404 returned error can't find the container with id 42b983056e413fcecf4a870c0e625f3c416431562cb77efaecdb154b372c0f17 Nov 01 01:10:21.883524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.883493 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" Nov 01 01:10:21.883524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.883518 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n" event={"ID":"2d8669ca-2949-4edd-8706-815cfe7055a0","Type":"ContainerDied","Data":"18001d99ef90cfd6c5d0c5df4e4d9ef91e2e4f7c5c5b53b126d0de25a9716dea"} Nov 01 01:10:21.883763 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.883567 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18001d99ef90cfd6c5d0c5df4e4d9ef91e2e4f7c5c5b53b126d0de25a9716dea" Nov 01 01:10:21.885091 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.885069 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" event={"ID":"339fac2d-201a-4c40-b722-366e74518ce4","Type":"ContainerStarted","Data":"586f58a2482a6fa429084a89d583a19f93a7c8d12c20ed9e560e3665cdf049c5"} Nov 01 01:10:21.885465 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:21.885108 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" event={"ID":"339fac2d-201a-4c40-b722-366e74518ce4","Type":"ContainerStarted","Data":"42b983056e413fcecf4a870c0e625f3c416431562cb77efaecdb154b372c0f17"} Nov 01 01:10:22.010554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:22.010506 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" podStartSLOduration=2.010491432 podStartE2EDuration="2.010491432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:10:22.00966769 +0000 UTC m=+2811.989250026" watchObservedRunningTime="2026-11-01 01:10:22.010491432 +0000 UTC m=+2811.990073797" Nov 01 01:10:22.895569 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:22.895528 4910 generic.go:356] "Generic (PLEG): container finished" podID="339fac2d-201a-4c40-b722-366e74518ce4" containerID="586f58a2482a6fa429084a89d583a19f93a7c8d12c20ed9e560e3665cdf049c5" exitCode=0 Nov 01 01:10:22.896088 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:22.895657 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" event={"ID":"339fac2d-201a-4c40-b722-366e74518ce4","Type":"ContainerDied","Data":"586f58a2482a6fa429084a89d583a19f93a7c8d12c20ed9e560e3665cdf049c5"} Nov 01 01:10:24.064606 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.064575 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:24.119400 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.119376 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339fac2d-201a-4c40-b722-366e74518ce4-kube-api-access-j2j49\" (UniqueName: \"kubernetes.io/projected/339fac2d-201a-4c40-b722-366e74518ce4-kube-api-access-j2j49\") pod \"339fac2d-201a-4c40-b722-366e74518ce4\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " Nov 01 01:10:24.119609 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.119587 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-xks-ca\") pod \"339fac2d-201a-4c40-b722-366e74518ce4\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " Nov 01 01:10:24.119745 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.119721 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-openshift-ca\") pod \"339fac2d-201a-4c40-b722-366e74518ce4\" (UID: \"339fac2d-201a-4c40-b722-366e74518ce4\") " Nov 01 01:10:24.120074 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.120049 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-openshift-ca" pod "339fac2d-201a-4c40-b722-366e74518ce4" (UID: "339fac2d-201a-4c40-b722-366e74518ce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:10:24.121666 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.121637 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339fac2d-201a-4c40-b722-366e74518ce4-kube-api-access-j2j49" pod "339fac2d-201a-4c40-b722-366e74518ce4" (UID: "339fac2d-201a-4c40-b722-366e74518ce4"). InnerVolumeSpecName "kube-api-access-j2j49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:24.121742 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.121672 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-xks-ca" pod "339fac2d-201a-4c40-b722-366e74518ce4" (UID: "339fac2d-201a-4c40-b722-366e74518ce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:10:24.220658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.220614 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:24.220658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.220658 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2j49\" (UniqueName: \"kubernetes.io/projected/339fac2d-201a-4c40-b722-366e74518ce4-kube-api-access-j2j49\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:24.220828 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.220674 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339fac2d-201a-4c40-b722-366e74518ce4-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:24.911321 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.911288 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" Nov 01 01:10:24.911515 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.911327 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs" event={"ID":"339fac2d-201a-4c40-b722-366e74518ce4","Type":"ContainerDied","Data":"42b983056e413fcecf4a870c0e625f3c416431562cb77efaecdb154b372c0f17"} Nov 01 01:10:24.911515 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:24.911374 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b983056e413fcecf4a870c0e625f3c416431562cb77efaecdb154b372c0f17" Nov 01 01:10:26.165336 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.165292 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc"] Nov 01 01:10:26.166144 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.166122 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2d8669ca-2949-4edd-8706-815cfe7055a0" containerName="revoke-keys" Nov 01 01:10:26.166144 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.166145 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8669ca-2949-4edd-8706-815cfe7055a0" containerName="revoke-keys" Nov 01 01:10:26.166312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.166160 4910 cpu_manager.go:414] "containerMap: removing container" podUID="339fac2d-201a-4c40-b722-366e74518ce4" containerName="revoke-keys" Nov 01 01:10:26.166312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.166167 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339fac2d-201a-4c40-b722-366e74518ce4" containerName="revoke-keys" Nov 01 01:10:26.166312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.166280 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8669ca-2949-4edd-8706-815cfe7055a0" containerName="revoke-keys" Nov 01 01:10:26.166312 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.166294 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="339fac2d-201a-4c40-b722-366e74518ce4" containerName="revoke-keys" Nov 01 01:10:26.170383 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.170362 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.224093 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.224069 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:10:26.224370 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.224355 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:10:26.254450 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.254420 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.254612 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.254558 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/ca8de625-5dfd-4925-b951-9dd507a1af4f-kube-api-access-q6xbs\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.254806 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.254787 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.278081 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.278055 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc"] Nov 01 01:10:26.356089 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.356060 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.356264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.356120 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.356264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.356199 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/ca8de625-5dfd-4925-b951-9dd507a1af4f-kube-api-access-q6xbs\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.356926 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.356864 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.358698 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.358677 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.419392 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.419328 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/ca8de625-5dfd-4925-b951-9dd507a1af4f-kube-api-access-q6xbs\") pod \"maas-api-revoke-keys-e2e-api-b-030da8-48ktc\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.488702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.488675 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:26.896253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.896219 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc"] Nov 01 01:10:26.896715 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:10:26.896692 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8de625_5dfd_4925_b951_9dd507a1af4f.slice/crio-867c0602680d80b4986b6dc14e83dde3b06d9bba11b59d08adb2308f18b40630 WatchSource:0}: Error finding container 867c0602680d80b4986b6dc14e83dde3b06d9bba11b59d08adb2308f18b40630: Status 404 returned error can't find the container with id 867c0602680d80b4986b6dc14e83dde3b06d9bba11b59d08adb2308f18b40630 Nov 01 01:10:26.926976 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:26.926948 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" event={"ID":"ca8de625-5dfd-4925-b951-9dd507a1af4f","Type":"ContainerStarted","Data":"867c0602680d80b4986b6dc14e83dde3b06d9bba11b59d08adb2308f18b40630"} Nov 01 01:10:27.936214 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:27.936181 4910 generic.go:356] "Generic (PLEG): container finished" podID="ca8de625-5dfd-4925-b951-9dd507a1af4f" containerID="d27b57c2df759a2605a9d1cebd353de69d4fbb7892b4eb6939239c6f7f03fe80" exitCode=0 Nov 01 01:10:27.936607 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:27.936266 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" event={"ID":"ca8de625-5dfd-4925-b951-9dd507a1af4f","Type":"ContainerDied","Data":"d27b57c2df759a2605a9d1cebd353de69d4fbb7892b4eb6939239c6f7f03fe80"} Nov 01 01:10:29.101778 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.101756 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:29.122969 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.122942 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-openshift-ca\") pod \"ca8de625-5dfd-4925-b951-9dd507a1af4f\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " Nov 01 01:10:29.123102 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.122979 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-xks-ca\") pod \"ca8de625-5dfd-4925-b951-9dd507a1af4f\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " Nov 01 01:10:29.123102 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.123001 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8de625-5dfd-4925-b951-9dd507a1af4f-kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/ca8de625-5dfd-4925-b951-9dd507a1af4f-kube-api-access-q6xbs\") pod \"ca8de625-5dfd-4925-b951-9dd507a1af4f\" (UID: \"ca8de625-5dfd-4925-b951-9dd507a1af4f\") " Nov 01 01:10:29.123362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.123330 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-openshift-ca" pod "ca8de625-5dfd-4925-b951-9dd507a1af4f" (UID: "ca8de625-5dfd-4925-b951-9dd507a1af4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:10:29.125127 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.125102 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-xks-ca" pod "ca8de625-5dfd-4925-b951-9dd507a1af4f" (UID: "ca8de625-5dfd-4925-b951-9dd507a1af4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:10:29.125222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.125172 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8de625-5dfd-4925-b951-9dd507a1af4f-kube-api-access-q6xbs" pod "ca8de625-5dfd-4925-b951-9dd507a1af4f" (UID: "ca8de625-5dfd-4925-b951-9dd507a1af4f"). InnerVolumeSpecName "kube-api-access-q6xbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:29.224231 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.224153 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:29.224231 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.224177 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8de625-5dfd-4925-b951-9dd507a1af4f-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:29.224231 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.224192 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/ca8de625-5dfd-4925-b951-9dd507a1af4f-kube-api-access-q6xbs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:29.953140 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.953111 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" Nov 01 01:10:29.953297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.953151 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc" event={"ID":"ca8de625-5dfd-4925-b951-9dd507a1af4f","Type":"ContainerDied","Data":"867c0602680d80b4986b6dc14e83dde3b06d9bba11b59d08adb2308f18b40630"} Nov 01 01:10:29.953297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:29.953203 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="867c0602680d80b4986b6dc14e83dde3b06d9bba11b59d08adb2308f18b40630" Nov 01 01:10:32.332514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.332472 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz"] Nov 01 01:10:32.333395 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.333365 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ca8de625-5dfd-4925-b951-9dd507a1af4f" containerName="revoke-keys" Nov 01 01:10:32.333395 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.333392 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8de625-5dfd-4925-b951-9dd507a1af4f" containerName="revoke-keys" Nov 01 01:10:32.333651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.333535 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8de625-5dfd-4925-b951-9dd507a1af4f" containerName="revoke-keys" Nov 01 01:10:32.337213 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.337191 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.359651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.359590 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.359792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.359653 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.359792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.359754 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscp9\" (UniqueName: \"kubernetes.io/projected/2bdfe58a-100b-4d95-9b77-3855753e0e36-kube-api-access-dscp9\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.400184 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.400153 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz"] Nov 01 01:10:32.444513 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.444484 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:10:32.444662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.444541 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:10:32.460570 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.460539 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscp9\" (UniqueName: \"kubernetes.io/projected/2bdfe58a-100b-4d95-9b77-3855753e0e36-kube-api-access-dscp9\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.460755 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.460736 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.460811 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.460769 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.461468 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.461447 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.463348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.463330 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.522332 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.522299 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscp9\" (UniqueName: \"kubernetes.io/projected/2bdfe58a-100b-4d95-9b77-3855753e0e36-kube-api-access-dscp9\") pod \"maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.651422 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.651352 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:32.863020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.862989 4910 scope.go:122] "RemoveContainer" containerID="41b0d1c8c0903dae110f820bf5bac2c445e6b81899e765fe59c08280f97a6184" Nov 01 01:10:32.874616 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:32.874587 4910 scope.go:122] "RemoveContainer" containerID="64f9336e6f1bcc5047cd16c943d9396a7d4b598df4280bf07796bd0732687b64" Nov 01 01:10:33.031695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:33.031673 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz"] Nov 01 01:10:33.032579 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:10:33.032551 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdfe58a_100b_4d95_9b77_3855753e0e36.slice/crio-daaff7606b192e7fd55eac12009aae052745b9cb5b93a7157b753a807e12be85 WatchSource:0}: Error finding container daaff7606b192e7fd55eac12009aae052745b9cb5b93a7157b753a807e12be85: Status 404 returned error can't find the container with id daaff7606b192e7fd55eac12009aae052745b9cb5b93a7157b753a807e12be85 Nov 01 01:10:33.984927 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:33.984892 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" event={"ID":"2bdfe58a-100b-4d95-9b77-3855753e0e36","Type":"ContainerStarted","Data":"dfbdfd7b185f675517dcb96611081401282527d82a4c519cff069484ddb107bf"} Nov 01 01:10:33.985346 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:33.984941 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" event={"ID":"2bdfe58a-100b-4d95-9b77-3855753e0e36","Type":"ContainerStarted","Data":"daaff7606b192e7fd55eac12009aae052745b9cb5b93a7157b753a807e12be85"} Nov 01 01:10:34.102665 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:34.102591 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" podStartSLOduration=2.102576637 podStartE2EDuration="2.102576637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:10:34.100754775 +0000 UTC m=+2824.080337154" watchObservedRunningTime="2026-11-01 01:10:34.102576637 +0000 UTC m=+2824.082159015" Nov 01 01:10:34.993502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:34.993466 4910 generic.go:356] "Generic (PLEG): container finished" podID="2bdfe58a-100b-4d95-9b77-3855753e0e36" containerID="dfbdfd7b185f675517dcb96611081401282527d82a4c519cff069484ddb107bf" exitCode=0 Nov 01 01:10:34.993934 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:34.993514 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" event={"ID":"2bdfe58a-100b-4d95-9b77-3855753e0e36","Type":"ContainerDied","Data":"dfbdfd7b185f675517dcb96611081401282527d82a4c519cff069484ddb107bf"} Nov 01 01:10:36.144837 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.144810 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:36.199379 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.199347 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-xks-ca\") pod \"2bdfe58a-100b-4d95-9b77-3855753e0e36\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " Nov 01 01:10:36.199529 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.199443 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdfe58a-100b-4d95-9b77-3855753e0e36-kube-api-access-dscp9\" (UniqueName: \"kubernetes.io/projected/2bdfe58a-100b-4d95-9b77-3855753e0e36-kube-api-access-dscp9\") pod \"2bdfe58a-100b-4d95-9b77-3855753e0e36\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " Nov 01 01:10:36.199529 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.199488 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-openshift-ca\") pod \"2bdfe58a-100b-4d95-9b77-3855753e0e36\" (UID: \"2bdfe58a-100b-4d95-9b77-3855753e0e36\") " Nov 01 01:10:36.200160 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.200136 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-openshift-ca" pod "2bdfe58a-100b-4d95-9b77-3855753e0e36" (UID: "2bdfe58a-100b-4d95-9b77-3855753e0e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:10:36.202302 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.202276 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-xks-ca" pod "2bdfe58a-100b-4d95-9b77-3855753e0e36" (UID: "2bdfe58a-100b-4d95-9b77-3855753e0e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:10:36.202490 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.202471 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdfe58a-100b-4d95-9b77-3855753e0e36-kube-api-access-dscp9" pod "2bdfe58a-100b-4d95-9b77-3855753e0e36" (UID: "2bdfe58a-100b-4d95-9b77-3855753e0e36"). InnerVolumeSpecName "kube-api-access-dscp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:36.300883 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.300810 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:36.300883 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.300837 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscp9\" (UniqueName: \"kubernetes.io/projected/2bdfe58a-100b-4d95-9b77-3855753e0e36-kube-api-access-dscp9\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:36.300883 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.300869 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdfe58a-100b-4d95-9b77-3855753e0e36-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:36.788715 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:36.788680 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt"] Nov 01 01:10:37.010153 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:37.010121 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" event={"ID":"2bdfe58a-100b-4d95-9b77-3855753e0e36","Type":"ContainerDied","Data":"daaff7606b192e7fd55eac12009aae052745b9cb5b93a7157b753a807e12be85"} Nov 01 01:10:37.010153 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:37.010151 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz" Nov 01 01:10:37.010369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:37.010170 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daaff7606b192e7fd55eac12009aae052745b9cb5b93a7157b753a807e12be85" Nov 01 01:10:43.120861 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:10:43.120778 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:10:44.141268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:44.141232 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz"] Nov 01 01:10:45.994373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:45.994331 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7"] Nov 01 01:10:46.044489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:46.044457 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-718e19-wbzl7"] Nov 01 01:10:46.192348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:46.192318 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04426f20-1fc6-453e-8bad-72097fed7296" path="/var/lib/kubelet/pods/04426f20-1fc6-453e-8bad-72097fed7296/volumes" Nov 01 01:10:49.142398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.142349 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" podUID="aa22de41-ac1f-4749-bf78-635a25a09efc" containerName="payload-pre-processing" containerID="cri-o://3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c" gracePeriod=25 Nov 01 01:10:49.398481 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.398409 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 01:10:49.501018 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.500980 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa22de41-ac1f-4749-bf78-635a25a09efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa22de41-ac1f-4749-bf78-635a25a09efc-plugins-config-volume\") pod \"aa22de41-ac1f-4749-bf78-635a25a09efc\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " Nov 01 01:10:49.501192 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.501108 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa22de41-ac1f-4749-bf78-635a25a09efc-kube-api-access-nxn2x\" (UniqueName: \"kubernetes.io/projected/aa22de41-ac1f-4749-bf78-635a25a09efc-kube-api-access-nxn2x\") pod \"aa22de41-ac1f-4749-bf78-635a25a09efc\" (UID: \"aa22de41-ac1f-4749-bf78-635a25a09efc\") " Nov 01 01:10:49.501377 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.501353 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa22de41-ac1f-4749-bf78-635a25a09efc-plugins-config-volume" pod "aa22de41-ac1f-4749-bf78-635a25a09efc" (UID: "aa22de41-ac1f-4749-bf78-635a25a09efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:10:49.501540 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.501515 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa22de41-ac1f-4749-bf78-635a25a09efc-plugins-config-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:49.503482 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.503454 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa22de41-ac1f-4749-bf78-635a25a09efc-kube-api-access-nxn2x" pod "aa22de41-ac1f-4749-bf78-635a25a09efc" (UID: "aa22de41-ac1f-4749-bf78-635a25a09efc"). InnerVolumeSpecName "kube-api-access-nxn2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:49.603007 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:49.602969 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxn2x\" (UniqueName: \"kubernetes.io/projected/aa22de41-ac1f-4749-bf78-635a25a09efc-kube-api-access-nxn2x\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:50.110435 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.110399 4910 generic.go:356] "Generic (PLEG): container finished" podID="aa22de41-ac1f-4749-bf78-635a25a09efc" containerID="3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c" exitCode=0 Nov 01 01:10:50.110686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.110451 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" Nov 01 01:10:50.110686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.110487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" event={"ID":"aa22de41-ac1f-4749-bf78-635a25a09efc","Type":"ContainerDied","Data":"3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c"} Nov 01 01:10:50.110686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.110579 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz" event={"ID":"aa22de41-ac1f-4749-bf78-635a25a09efc","Type":"ContainerDied","Data":"0e8b9354b006eb21b61256f87fa59d19ede31817563dc469517b93ba72e9ce86"} Nov 01 01:10:50.110686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.110609 4910 scope.go:122] "RemoveContainer" containerID="3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c" Nov 01 01:10:50.122027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.122011 4910 scope.go:122] "RemoveContainer" containerID="3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c" Nov 01 01:10:50.122293 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:10:50.122275 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c\": container with ID starting with 3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c not found: ID does not exist" containerID="3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c" Nov 01 01:10:50.122336 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.122301 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c"} err="failed to get container status \"3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c\": rpc error: code = NotFound desc = could not find container \"3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c\": container with ID starting with 3cde836b0aeeea269044d261b3145f030757f4e9c7ead5ec64394875442f2a8c not found: ID does not exist" Nov 01 01:10:50.288769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.288729 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz"] Nov 01 01:10:50.339133 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:50.339102 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5a9c19-556998b48c-jt2nz"] Nov 01 01:10:51.810366 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:51.810321 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" containerID="cri-o://801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490" gracePeriod=45 Nov 01 01:10:52.078593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.078536 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 01:10:52.126816 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.126787 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-model-cache\") pod \"866123ba-31f1-4403-bdd1-53a699b2d4f9\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " Nov 01 01:10:52.126983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.126836 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866123ba-31f1-4403-bdd1-53a699b2d4f9-kube-api-access-9gmgs\" (UniqueName: \"kubernetes.io/projected/866123ba-31f1-4403-bdd1-53a699b2d4f9-kube-api-access-9gmgs\") pod \"866123ba-31f1-4403-bdd1-53a699b2d4f9\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " Nov 01 01:10:52.126983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.126887 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/866123ba-31f1-4403-bdd1-53a699b2d4f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/866123ba-31f1-4403-bdd1-53a699b2d4f9-tls-certs\") pod \"866123ba-31f1-4403-bdd1-53a699b2d4f9\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " Nov 01 01:10:52.126983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.126967 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-home\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-home\") pod \"866123ba-31f1-4403-bdd1-53a699b2d4f9\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " Nov 01 01:10:52.127151 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.127021 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-tmp-dir\") pod \"866123ba-31f1-4403-bdd1-53a699b2d4f9\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " Nov 01 01:10:52.127151 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.127067 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-model-cache" pod "866123ba-31f1-4403-bdd1-53a699b2d4f9" (UID: "866123ba-31f1-4403-bdd1-53a699b2d4f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.127254 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.127222 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-home" pod "866123ba-31f1-4403-bdd1-53a699b2d4f9" (UID: "866123ba-31f1-4403-bdd1-53a699b2d4f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.127337 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.127316 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-tmp-dir" pod "866123ba-31f1-4403-bdd1-53a699b2d4f9" (UID: "866123ba-31f1-4403-bdd1-53a699b2d4f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.127507 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.127478 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.127507 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.127497 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.127696 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.127510 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.128671 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.128615 4910 generic.go:356] "Generic (PLEG): container finished" podID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerID="801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490" exitCode=0 Nov 01 01:10:52.128801 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.128698 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" Nov 01 01:10:52.128801 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.128698 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" event={"ID":"866123ba-31f1-4403-bdd1-53a699b2d4f9","Type":"ContainerDied","Data":"801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490"} Nov 01 01:10:52.128801 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.128741 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt" event={"ID":"866123ba-31f1-4403-bdd1-53a699b2d4f9","Type":"ContainerDied","Data":"9754cc2afe0228ef8934fad4154479d514ed04a4866b1e07e71c97801da8eeab"} Nov 01 01:10:52.128801 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.128760 4910 scope.go:122] "RemoveContainer" containerID="801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490" Nov 01 01:10:52.129955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.129937 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/866123ba-31f1-4403-bdd1-53a699b2d4f9-tls-certs" pod "866123ba-31f1-4403-bdd1-53a699b2d4f9" (UID: "866123ba-31f1-4403-bdd1-53a699b2d4f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:10:52.130057 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.129990 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866123ba-31f1-4403-bdd1-53a699b2d4f9-kube-api-access-9gmgs" pod "866123ba-31f1-4403-bdd1-53a699b2d4f9" (UID: "866123ba-31f1-4403-bdd1-53a699b2d4f9"). InnerVolumeSpecName "kube-api-access-9gmgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:52.151086 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.151072 4910 scope.go:122] "RemoveContainer" containerID="801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490" Nov 01 01:10:52.151401 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:10:52.151371 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490\": container with ID starting with 801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490 not found: ID does not exist" containerID="801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490" Nov 01 01:10:52.151516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.151411 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490"} err="failed to get container status \"801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490\": rpc error: code = NotFound desc = could not find container \"801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490\": container with ID starting with 801eaffb09de70455cf1c349c0ec29d57547bb42c1d0aa437342d66f029b5490 not found: ID does not exist" Nov 01 01:10:52.195538 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.195506 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa22de41-ac1f-4749-bf78-635a25a09efc" path="/var/lib/kubelet/pods/aa22de41-ac1f-4749-bf78-635a25a09efc/volumes" Nov 01 01:10:52.228268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.228237 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-dshm\") pod \"866123ba-31f1-4403-bdd1-53a699b2d4f9\" (UID: \"866123ba-31f1-4403-bdd1-53a699b2d4f9\") " Nov 01 01:10:52.228761 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.228733 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gmgs\" (UniqueName: \"kubernetes.io/projected/866123ba-31f1-4403-bdd1-53a699b2d4f9-kube-api-access-9gmgs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.229084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.228798 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866123ba-31f1-4403-bdd1-53a699b2d4f9-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.230829 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.230795 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-dshm" pod "866123ba-31f1-4403-bdd1-53a699b2d4f9" (UID: "866123ba-31f1-4403-bdd1-53a699b2d4f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:10:52.329775 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.329694 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866123ba-31f1-4403-bdd1-53a699b2d4f9-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:52.552480 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.552440 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt"] Nov 01 01:10:52.558327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:52.558300 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5a9c19/simulator-w3-5a9c19-kserve-699dfbd894-87mzt"] Nov 01 01:10:54.191979 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.191942 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" path="/var/lib/kubelet/pods/866123ba-31f1-4403-bdd1-53a699b2d4f9/volumes" Nov 01 01:10:54.848286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848249 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf"] Nov 01 01:10:54.848821 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848806 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2bdfe58a-100b-4d95-9b77-3855753e0e36" containerName="revoke-keys" Nov 01 01:10:54.848873 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848825 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdfe58a-100b-4d95-9b77-3855753e0e36" containerName="revoke-keys" Nov 01 01:10:54.848873 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848859 4910 cpu_manager.go:414] "containerMap: removing container" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" Nov 01 01:10:54.848873 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848865 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" Nov 01 01:10:54.848981 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848879 4910 cpu_manager.go:414] "containerMap: removing container" podUID="aa22de41-ac1f-4749-bf78-635a25a09efc" containerName="payload-pre-processing" Nov 01 01:10:54.848981 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848885 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa22de41-ac1f-4749-bf78-635a25a09efc" containerName="payload-pre-processing" Nov 01 01:10:54.848981 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848956 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa22de41-ac1f-4749-bf78-635a25a09efc" containerName="payload-pre-processing" Nov 01 01:10:54.848981 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848968 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdfe58a-100b-4d95-9b77-3855753e0e36" containerName="revoke-keys" Nov 01 01:10:54.848981 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.848978 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="866123ba-31f1-4403-bdd1-53a699b2d4f9" containerName="main" Nov 01 01:10:54.853730 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.853711 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:54.912979 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.912938 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf"] Nov 01 01:10:54.955602 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.955564 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkptw\" (UniqueName: \"kubernetes.io/projected/4d592d80-3cab-4752-a7cb-96a621bd2216-kube-api-access-bkptw\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:54.955602 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.955606 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:54.955872 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.955713 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:54.973412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.973388 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:10:54.973548 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:54.973436 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:10:55.056952 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.056924 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:55.057120 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.057039 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkptw\" (UniqueName: \"kubernetes.io/projected/4d592d80-3cab-4752-a7cb-96a621bd2216-kube-api-access-bkptw\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:55.057120 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.057065 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:55.057770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.057696 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:55.059415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.059394 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:55.124720 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.124658 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkptw\" (UniqueName: \"kubernetes.io/projected/4d592d80-3cab-4752-a7cb-96a621bd2216-kube-api-access-bkptw\") pod \"maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:55.168821 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.168798 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:55.570103 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:55.570078 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf"] Nov 01 01:10:55.570800 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:10:55.570771 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d592d80_3cab_4752_a7cb_96a621bd2216.slice/crio-a1c11801cb09717b65501f65dc902dde3b3de9982ca204774f631abf6364485b WatchSource:0}: Error finding container a1c11801cb09717b65501f65dc902dde3b3de9982ca204774f631abf6364485b: Status 404 returned error can't find the container with id a1c11801cb09717b65501f65dc902dde3b3de9982ca204774f631abf6364485b Nov 01 01:10:56.161991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:56.161894 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" event={"ID":"4d592d80-3cab-4752-a7cb-96a621bd2216","Type":"ContainerStarted","Data":"93bd2e33e43f0087b568311a2080a547c6ff40a135e9157cb7110221e3df7257"} Nov 01 01:10:56.161991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:56.161960 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" event={"ID":"4d592d80-3cab-4752-a7cb-96a621bd2216","Type":"ContainerStarted","Data":"a1c11801cb09717b65501f65dc902dde3b3de9982ca204774f631abf6364485b"} Nov 01 01:10:57.171819 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:57.171772 4910 generic.go:356] "Generic (PLEG): container finished" podID="4d592d80-3cab-4752-a7cb-96a621bd2216" containerID="93bd2e33e43f0087b568311a2080a547c6ff40a135e9157cb7110221e3df7257" exitCode=0 Nov 01 01:10:57.172252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:57.171856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" event={"ID":"4d592d80-3cab-4752-a7cb-96a621bd2216","Type":"ContainerDied","Data":"93bd2e33e43f0087b568311a2080a547c6ff40a135e9157cb7110221e3df7257"} Nov 01 01:10:58.326697 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.326674 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:58.396259 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.396231 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-xks-ca\") pod \"4d592d80-3cab-4752-a7cb-96a621bd2216\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " Nov 01 01:10:58.396442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.396323 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d592d80-3cab-4752-a7cb-96a621bd2216-kube-api-access-bkptw\" (UniqueName: \"kubernetes.io/projected/4d592d80-3cab-4752-a7cb-96a621bd2216-kube-api-access-bkptw\") pod \"4d592d80-3cab-4752-a7cb-96a621bd2216\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " Nov 01 01:10:58.396442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.396428 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-openshift-ca\") pod \"4d592d80-3cab-4752-a7cb-96a621bd2216\" (UID: \"4d592d80-3cab-4752-a7cb-96a621bd2216\") " Nov 01 01:10:58.396825 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.396793 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-openshift-ca" pod "4d592d80-3cab-4752-a7cb-96a621bd2216" (UID: "4d592d80-3cab-4752-a7cb-96a621bd2216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:10:58.398647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.398594 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-xks-ca" pod "4d592d80-3cab-4752-a7cb-96a621bd2216" (UID: "4d592d80-3cab-4752-a7cb-96a621bd2216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:10:58.398785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.398709 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d592d80-3cab-4752-a7cb-96a621bd2216-kube-api-access-bkptw" pod "4d592d80-3cab-4752-a7cb-96a621bd2216" (UID: "4d592d80-3cab-4752-a7cb-96a621bd2216"). InnerVolumeSpecName "kube-api-access-bkptw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:10:58.498142 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.498114 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkptw\" (UniqueName: \"kubernetes.io/projected/4d592d80-3cab-4752-a7cb-96a621bd2216-kube-api-access-bkptw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:58.498324 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.498145 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:58.498324 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:58.498162 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d592d80-3cab-4752-a7cb-96a621bd2216-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:10:59.187859 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:59.187818 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" event={"ID":"4d592d80-3cab-4752-a7cb-96a621bd2216","Type":"ContainerDied","Data":"a1c11801cb09717b65501f65dc902dde3b3de9982ca204774f631abf6364485b"} Nov 01 01:10:59.187859 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:59.187844 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf" Nov 01 01:10:59.188125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:10:59.187875 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1c11801cb09717b65501f65dc902dde3b3de9982ca204774f631abf6364485b" Nov 01 01:11:11.586978 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.586943 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr"] Nov 01 01:11:11.587483 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.587418 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4d592d80-3cab-4752-a7cb-96a621bd2216" containerName="revoke-keys" Nov 01 01:11:11.587483 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.587429 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d592d80-3cab-4752-a7cb-96a621bd2216" containerName="revoke-keys" Nov 01 01:11:11.587608 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.587527 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d592d80-3cab-4752-a7cb-96a621bd2216" containerName="revoke-keys" Nov 01 01:11:11.590926 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.590903 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.644280 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.644255 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rd62\" (UniqueName: \"kubernetes.io/projected/2a51e047-9bbd-450c-898b-86c9e79781bd-kube-api-access-5rd62\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.644601 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.644291 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.644601 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.644414 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.645740 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.645713 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:11:11.650886 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.650859 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr"] Nov 01 01:11:11.695952 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.695928 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:11:11.745699 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.745665 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rd62\" (UniqueName: \"kubernetes.io/projected/2a51e047-9bbd-450c-898b-86c9e79781bd-kube-api-access-5rd62\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.745879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.745713 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.745879 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.745846 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.747152 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.747126 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.748847 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.748825 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.807831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.807798 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rd62\" (UniqueName: \"kubernetes.io/projected/2a51e047-9bbd-450c-898b-86c9e79781bd-kube-api-access-5rd62\") pod \"maas-api-revoke-sub-7579f87a-6lzxr\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:11.907514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:11.907444 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:12.290930 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:11:12.290897 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a51e047_9bbd_450c_898b_86c9e79781bd.slice/crio-e5b6f0e4f56f043ef296cde820164aba2d0dd74a2af7d3a1eb44ffcd429e9466 WatchSource:0}: Error finding container e5b6f0e4f56f043ef296cde820164aba2d0dd74a2af7d3a1eb44ffcd429e9466: Status 404 returned error can't find the container with id e5b6f0e4f56f043ef296cde820164aba2d0dd74a2af7d3a1eb44ffcd429e9466 Nov 01 01:11:12.291521 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:12.291493 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr"] Nov 01 01:11:12.309555 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:12.309399 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" event={"ID":"2a51e047-9bbd-450c-898b-86c9e79781bd","Type":"ContainerStarted","Data":"e5b6f0e4f56f043ef296cde820164aba2d0dd74a2af7d3a1eb44ffcd429e9466"} Nov 01 01:11:13.121542 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:11:13.121499 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:11:13.331768 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:13.330987 4910 generic.go:356] "Generic (PLEG): container finished" podID="2a51e047-9bbd-450c-898b-86c9e79781bd" containerID="a906b8ca8d03cdfe1350abec331e996fca1098e16260ba65ce220a0e6319d6f4" exitCode=0 Nov 01 01:11:13.331768 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:13.331427 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" event={"ID":"2a51e047-9bbd-450c-898b-86c9e79781bd","Type":"ContainerDied","Data":"a906b8ca8d03cdfe1350abec331e996fca1098e16260ba65ce220a0e6319d6f4"} Nov 01 01:11:14.487225 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.487198 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:14.577772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.577736 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-xks-ca\") pod \"2a51e047-9bbd-450c-898b-86c9e79781bd\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " Nov 01 01:11:14.577961 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.577795 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a51e047-9bbd-450c-898b-86c9e79781bd-kube-api-access-5rd62\" (UniqueName: \"kubernetes.io/projected/2a51e047-9bbd-450c-898b-86c9e79781bd-kube-api-access-5rd62\") pod \"2a51e047-9bbd-450c-898b-86c9e79781bd\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " Nov 01 01:11:14.578014 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.577977 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-openshift-ca\") pod \"2a51e047-9bbd-450c-898b-86c9e79781bd\" (UID: \"2a51e047-9bbd-450c-898b-86c9e79781bd\") " Nov 01 01:11:14.578376 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.578339 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-openshift-ca" pod "2a51e047-9bbd-450c-898b-86c9e79781bd" (UID: "2a51e047-9bbd-450c-898b-86c9e79781bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:14.580702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.580675 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-xks-ca" pod "2a51e047-9bbd-450c-898b-86c9e79781bd" (UID: "2a51e047-9bbd-450c-898b-86c9e79781bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:11:14.580809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.580712 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a51e047-9bbd-450c-898b-86c9e79781bd-kube-api-access-5rd62" pod "2a51e047-9bbd-450c-898b-86c9e79781bd" (UID: "2a51e047-9bbd-450c-898b-86c9e79781bd"). InnerVolumeSpecName "kube-api-access-5rd62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:14.679460 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.679373 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:14.679460 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.679405 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51e047-9bbd-450c-898b-86c9e79781bd-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:14.679460 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:14.679416 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rd62\" (UniqueName: \"kubernetes.io/projected/2a51e047-9bbd-450c-898b-86c9e79781bd-kube-api-access-5rd62\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:15.346530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:15.346492 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" event={"ID":"2a51e047-9bbd-450c-898b-86c9e79781bd","Type":"ContainerDied","Data":"e5b6f0e4f56f043ef296cde820164aba2d0dd74a2af7d3a1eb44ffcd429e9466"} Nov 01 01:11:15.346530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:15.346534 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr" Nov 01 01:11:15.346785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:15.346552 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5b6f0e4f56f043ef296cde820164aba2d0dd74a2af7d3a1eb44ffcd429e9466" Nov 01 01:11:19.767609 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:19.767579 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n"] Nov 01 01:11:19.815297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:19.815264 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9b61-6k57n"] Nov 01 01:11:20.193755 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:20.193676 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8669ca-2949-4edd-8706-815cfe7055a0" path="/var/lib/kubelet/pods/2d8669ca-2949-4edd-8706-815cfe7055a0/volumes" Nov 01 01:11:32.909492 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:32.909461 4910 scope.go:122] "RemoveContainer" containerID="35dd2d3a62cdd7746ce61bb6f2abc80fabe17b9fcf5a92d427add5f2526fb6f7" Nov 01 01:11:32.924852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:32.924831 4910 scope.go:122] "RemoveContainer" containerID="353328333bf875195a25a751996773934c6aeaadec14984b22a719626ccfc0ad" Nov 01 01:11:32.934395 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:32.934370 4910 scope.go:122] "RemoveContainer" containerID="0d15c88a5ffa84bc2d9da46631e66d1207db123294d47cb6e5e81250d23f4845" Nov 01 01:11:42.619853 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:42.619811 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc"] Nov 01 01:11:42.623150 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:42.623107 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-030da8-48ktc"] Nov 01 01:11:43.103190 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:11:43.103149 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:11:44.191805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:44.191769 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8de625-5dfd-4925-b951-9dd507a1af4f" path="/var/lib/kubelet/pods/ca8de625-5dfd-4925-b951-9dd507a1af4f/volumes" Nov 01 01:11:47.024647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:47.024150 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5"] Nov 01 01:11:47.024647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:47.024402 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" containerID="cri-o://e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0" gracePeriod=30 Nov 01 01:11:49.046521 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:49.046487 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz"] Nov 01 01:11:49.096442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:49.096414 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5a9c19-jkrqz"] Nov 01 01:11:50.193485 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:50.193448 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdfe58a-100b-4d95-9b77-3855753e0e36" path="/var/lib/kubelet/pods/2bdfe58a-100b-4d95-9b77-3855753e0e36/volumes" Nov 01 01:11:52.295872 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.295811 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 01:11:52.438261 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438231 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-credential-socket\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438286 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-certs\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438321 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-kube-api-access-k85m4\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-kube-api-access-k85m4\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438381 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d85d550-af31-494e-aee4-d30f9b0900e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d85d550-af31-494e-aee4-d30f9b0900e5-istiod-ca-cert\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438412 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-podinfo\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438455 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-data\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438504 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-socket\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438534 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-credential-socket" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:52.438702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438538 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-token\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-token\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438669 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-certs" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:52.438990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438747 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-envoy\") pod \"4d85d550-af31-494e-aee4-d30f9b0900e5\" (UID: \"4d85d550-af31-494e-aee4-d30f9b0900e5\") " Nov 01 01:11:52.438990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438790 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d85d550-af31-494e-aee4-d30f9b0900e5-istiod-ca-cert" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:11:52.438990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.438893 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-data" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:52.439163 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.439055 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-socket" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:52.439233 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.439188 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d85d550-af31-494e-aee4-d30f9b0900e5-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.439449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.439332 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.439449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.439389 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.439449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.439402 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.439449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.439415 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.441018 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.440993 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-kube-api-access-k85m4" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "kube-api-access-k85m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:52.441144 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.441030 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-token" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:11:52.441144 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.441068 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-envoy" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:11:52.441228 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.441161 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-podinfo" pod "4d85d550-af31-494e-aee4-d30f9b0900e5" (UID: "4d85d550-af31-494e-aee4-d30f9b0900e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:11:52.540725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.540697 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.540725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.540722 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k85m4\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-kube-api-access-k85m4\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.540725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.540733 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.540961 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.540742 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d85d550-af31-494e-aee4-d30f9b0900e5-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:11:52.732506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.732426 4910 generic.go:356] "Generic (PLEG): container finished" podID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerID="e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0" exitCode=0 Nov 01 01:11:52.732506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.732471 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" Nov 01 01:11:52.732766 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.732519 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" event={"ID":"4d85d550-af31-494e-aee4-d30f9b0900e5","Type":"ContainerDied","Data":"e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0"} Nov 01 01:11:52.732766 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.732573 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5" event={"ID":"4d85d550-af31-494e-aee4-d30f9b0900e5","Type":"ContainerDied","Data":"309bd62be4e00984a10536facc400339206f9098d8cbe495d92703be1ec3d73a"} Nov 01 01:11:52.732766 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.732600 4910 scope.go:122] "RemoveContainer" containerID="e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0" Nov 01 01:11:52.745222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.745204 4910 scope.go:122] "RemoveContainer" containerID="e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0" Nov 01 01:11:52.745471 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:11:52.745454 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0\": container with ID starting with e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0 not found: ID does not exist" containerID="e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0" Nov 01 01:11:52.745520 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.745480 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0"} err="failed to get container status \"e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0\": rpc error: code = NotFound desc = could not find container \"e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0\": container with ID starting with e105a56e63f2903e097dad0f879a3394d1b53c60ec0f6268833cf9cc03ec10a0 not found: ID does not exist" Nov 01 01:11:52.910507 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.910477 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5"] Nov 01 01:11:52.920250 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:52.920222 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-43a0cf-openshift-default-6d8944d699-5d9s5"] Nov 01 01:11:54.194356 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:54.194317 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" path="/var/lib/kubelet/pods/4d85d550-af31-494e-aee4-d30f9b0900e5/volumes" Nov 01 01:11:55.558032 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:55.558001 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf"] Nov 01 01:11:55.561240 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:55.561217 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbea8a-f59bf"] Nov 01 01:11:56.193956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:56.193914 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d592d80-3cab-4752-a7cb-96a621bd2216" path="/var/lib/kubelet/pods/4d592d80-3cab-4752-a7cb-96a621bd2216/volumes" Nov 01 01:11:56.617609 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:56.617575 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx"] Nov 01 01:11:56.618135 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:11:56.617814 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" containerID="cri-o://ddf961362ce532e30700c934fdb01026f6016beae59a489eaf6f93b9e61094b2" gracePeriod=30 Nov 01 01:12:00.543587 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:00.543553 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:12:00.544102 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:00.543613 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:12:01.826709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.826391 4910 generic.go:356] "Generic (PLEG): container finished" podID="cff05228-9e12-407d-b097-afba164aab6c" containerID="ddf961362ce532e30700c934fdb01026f6016beae59a489eaf6f93b9e61094b2" exitCode=0 Nov 01 01:12:01.826709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.826604 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" event={"ID":"cff05228-9e12-407d-b097-afba164aab6c","Type":"ContainerDied","Data":"ddf961362ce532e30700c934fdb01026f6016beae59a489eaf6f93b9e61094b2"} Nov 01 01:12:01.921068 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.921038 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 01:12:01.959633 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959606 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-certs\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.959796 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959670 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff05228-9e12-407d-b097-afba164aab6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff05228-9e12-407d-b097-afba164aab6c-istiod-ca-cert\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.959796 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959713 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-istio-token\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-istio-token\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.959796 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959748 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cff05228-9e12-407d-b097-afba164aab6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff05228-9e12-407d-b097-afba164aab6c-istio-podinfo\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.959796 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959779 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-credential-socket\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.960034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959824 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-socket\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.960034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959876 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-kube-api-access-9rptv\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-kube-api-access-9rptv\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.960034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959902 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-certs" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:01.960034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.959966 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-data\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.960034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960006 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-envoy\") pod \"cff05228-9e12-407d-b097-afba164aab6c\" (UID: \"cff05228-9e12-407d-b097-afba164aab6c\") " Nov 01 01:12:01.960288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960074 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff05228-9e12-407d-b097-afba164aab6c-istiod-ca-cert" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:01.960288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960243 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-credential-socket" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:01.960288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960231 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-socket" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:01.960449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960281 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-data" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:01.960449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960431 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:01.960552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960450 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff05228-9e12-407d-b097-afba164aab6c-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:01.960552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960465 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:01.960552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960479 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:01.960552 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.960494 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:01.963016 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.962987 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-kube-api-access-9rptv" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "kube-api-access-9rptv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:01.963016 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.962996 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cff05228-9e12-407d-b097-afba164aab6c-istio-podinfo" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:12:01.963168 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.963081 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-envoy" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:01.963168 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:01.963128 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-istio-token" pod "cff05228-9e12-407d-b097-afba164aab6c" (UID: "cff05228-9e12-407d-b097-afba164aab6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:02.061913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:02.061878 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:02.061913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:02.061911 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff05228-9e12-407d-b097-afba164aab6c-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:02.062116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:02.061922 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rptv\" (UniqueName: \"kubernetes.io/projected/cff05228-9e12-407d-b097-afba164aab6c-kube-api-access-9rptv\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:02.062116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:02.061932 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff05228-9e12-407d-b097-afba164aab6c-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:02.834780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:02.834748 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" event={"ID":"cff05228-9e12-407d-b097-afba164aab6c","Type":"ContainerDied","Data":"ca7cfe2506a8f4c489a5ad16c8017bac4196765639fab1ffdf9d02444b565ee1"} Nov 01 01:12:02.834780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:02.834770 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx" Nov 01 01:12:02.835280 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:02.834813 4910 scope.go:122] "RemoveContainer" containerID="ddf961362ce532e30700c934fdb01026f6016beae59a489eaf6f93b9e61094b2" Nov 01 01:12:03.058298 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:03.058261 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx"] Nov 01 01:12:03.105718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:03.105617 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5a9c19-openshift-default-66fbfb6c95-pkzcx"] Nov 01 01:12:03.108740 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:03.108715 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf"] Nov 01 01:12:03.111539 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:03.111512 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8a1d22-2v7zf"] Nov 01 01:12:04.053061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.053003 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq"] Nov 01 01:12:04.060412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.060388 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4808cb9-jjrjq"] Nov 01 01:12:04.192542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.192504 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddcea4c-009d-4f8d-ae47-e06d0e265be9" path="/var/lib/kubelet/pods/4ddcea4c-009d-4f8d-ae47-e06d0e265be9/volumes" Nov 01 01:12:04.192953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.192938 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8291ba-087b-486d-a4b8-c4a3708f5a41" path="/var/lib/kubelet/pods/7a8291ba-087b-486d-a4b8-c4a3708f5a41/volumes" Nov 01 01:12:04.193362 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.193350 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff05228-9e12-407d-b097-afba164aab6c" path="/var/lib/kubelet/pods/cff05228-9e12-407d-b097-afba164aab6c/volumes" Nov 01 01:12:04.549034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549000 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8"] Nov 01 01:12:04.549497 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549483 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2a51e047-9bbd-450c-898b-86c9e79781bd" containerName="revoke-keys" Nov 01 01:12:04.549550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549498 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a51e047-9bbd-450c-898b-86c9e79781bd" containerName="revoke-keys" Nov 01 01:12:04.549550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549513 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" Nov 01 01:12:04.549550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549518 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" Nov 01 01:12:04.549550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549538 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" Nov 01 01:12:04.549550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549544 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" Nov 01 01:12:04.549782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549608 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff05228-9e12-407d-b097-afba164aab6c" containerName="istio-proxy" Nov 01 01:12:04.549782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549636 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a51e047-9bbd-450c-898b-86c9e79781bd" containerName="revoke-keys" Nov 01 01:12:04.549782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.549643 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d85d550-af31-494e-aee4-d30f9b0900e5" containerName="istio-proxy" Nov 01 01:12:04.554207 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.554189 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.606731 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.606699 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:12:04.611684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.611654 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8"] Nov 01 01:12:04.656477 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.656451 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:12:04.688663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.688608 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxn7g\" (UniqueName: \"kubernetes.io/projected/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-kube-api-access-fxn7g\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.688845 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.688785 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.688845 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.688837 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.790184 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.790151 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxn7g\" (UniqueName: \"kubernetes.io/projected/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-kube-api-access-fxn7g\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.790380 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.790221 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.790380 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.790259 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.790981 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.790960 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.792749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.792723 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.852236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.852152 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxn7g\" (UniqueName: \"kubernetes.io/projected/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-kube-api-access-fxn7g\") pod \"maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:04.867418 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:04.867393 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:05.249594 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:05.249572 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8"] Nov 01 01:12:05.251158 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:12:05.251130 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d71d4d0_0bec_4bfc_bf90_80c2aad63028.slice/crio-10ddad6958585359bc49f2d82e42cae867b62235240fd2d8a05c7dd8398900b6 WatchSource:0}: Error finding container 10ddad6958585359bc49f2d82e42cae867b62235240fd2d8a05c7dd8398900b6: Status 404 returned error can't find the container with id 10ddad6958585359bc49f2d82e42cae867b62235240fd2d8a05c7dd8398900b6 Nov 01 01:12:05.860092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:05.859988 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" event={"ID":"7d71d4d0-0bec-4bfc-bf90-80c2aad63028","Type":"ContainerStarted","Data":"a7fe3144439c76b38c1ec84476e8241dffed92e611519af35e23f71bb36e1131"} Nov 01 01:12:05.860272 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:05.860095 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" event={"ID":"7d71d4d0-0bec-4bfc-bf90-80c2aad63028","Type":"ContainerStarted","Data":"10ddad6958585359bc49f2d82e42cae867b62235240fd2d8a05c7dd8398900b6"} Nov 01 01:12:05.978391 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:05.978336 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" podStartSLOduration=1.97832136 podStartE2EDuration="1.97832136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:05.976593974 +0000 UTC m=+2915.956176310" watchObservedRunningTime="2026-11-01 01:12:05.97832136 +0000 UTC m=+2915.957903695" Nov 01 01:12:06.870437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:06.870402 4910 generic.go:356] "Generic (PLEG): container finished" podID="7d71d4d0-0bec-4bfc-bf90-80c2aad63028" containerID="a7fe3144439c76b38c1ec84476e8241dffed92e611519af35e23f71bb36e1131" exitCode=0 Nov 01 01:12:06.871015 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:06.870490 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" event={"ID":"7d71d4d0-0bec-4bfc-bf90-80c2aad63028","Type":"ContainerDied","Data":"a7fe3144439c76b38c1ec84476e8241dffed92e611519af35e23f71bb36e1131"} Nov 01 01:12:08.025277 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.025252 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:08.123916 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.123887 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-xks-ca\") pod \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " Nov 01 01:12:08.124070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.123954 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-openshift-ca\") pod \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " Nov 01 01:12:08.124070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.124038 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-kube-api-access-fxn7g\" (UniqueName: \"kubernetes.io/projected/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-kube-api-access-fxn7g\") pod \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\" (UID: \"7d71d4d0-0bec-4bfc-bf90-80c2aad63028\") " Nov 01 01:12:08.124316 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.124285 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-openshift-ca" pod "7d71d4d0-0bec-4bfc-bf90-80c2aad63028" (UID: "7d71d4d0-0bec-4bfc-bf90-80c2aad63028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:08.124436 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.124402 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:08.126226 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.126206 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-xks-ca" pod "7d71d4d0-0bec-4bfc-bf90-80c2aad63028" (UID: "7d71d4d0-0bec-4bfc-bf90-80c2aad63028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:08.126308 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.126250 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-kube-api-access-fxn7g" pod "7d71d4d0-0bec-4bfc-bf90-80c2aad63028" (UID: "7d71d4d0-0bec-4bfc-bf90-80c2aad63028"). InnerVolumeSpecName "kube-api-access-fxn7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:08.225177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.225143 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxn7g\" (UniqueName: \"kubernetes.io/projected/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-kube-api-access-fxn7g\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:08.225177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.225170 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71d4d0-0bec-4bfc-bf90-80c2aad63028-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:08.888413 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.888331 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" Nov 01 01:12:08.888413 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.888373 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8" event={"ID":"7d71d4d0-0bec-4bfc-bf90-80c2aad63028","Type":"ContainerDied","Data":"10ddad6958585359bc49f2d82e42cae867b62235240fd2d8a05c7dd8398900b6"} Nov 01 01:12:08.888609 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:08.888431 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ddad6958585359bc49f2d82e42cae867b62235240fd2d8a05c7dd8398900b6" Nov 01 01:12:13.103196 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:12:13.103157 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:12:16.219033 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:16.218997 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp"] Nov 01 01:12:21.220265 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:21.220221 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" podUID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" containerName="payload-pre-processing" containerID="cri-o://73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8" gracePeriod=25 Nov 01 01:12:22.164110 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.164078 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t"] Nov 01 01:12:22.164589 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.164576 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7d71d4d0-0bec-4bfc-bf90-80c2aad63028" containerName="revoke-keys" Nov 01 01:12:22.164669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.164590 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d71d4d0-0bec-4bfc-bf90-80c2aad63028" containerName="revoke-keys" Nov 01 01:12:22.164716 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.164693 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d71d4d0-0bec-4bfc-bf90-80c2aad63028" containerName="revoke-keys" Nov 01 01:12:22.168194 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.168178 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.225112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.225077 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp"] Nov 01 01:12:22.229224 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.229203 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.271412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.271389 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:12:22.274070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.274039 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:12:22.274821 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.274800 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t"] Nov 01 01:12:22.277487 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.277469 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.277819 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.277792 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xj8v\" (UniqueName: \"kubernetes.io/projected/1992a42f-a61e-4234-b4c0-2d1a21494785-kube-api-access-4xj8v\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.277936 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.277857 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.286152 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.286132 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp"] Nov 01 01:12:22.379373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.379339 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq5br\" (UniqueName: \"kubernetes.io/projected/ef7324e8-e1af-4561-aa01-6fcee3cb4717-kube-api-access-zq5br\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.379550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.379389 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.379550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.379499 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.379550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.379533 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xj8v\" (UniqueName: \"kubernetes.io/projected/1992a42f-a61e-4234-b4c0-2d1a21494785-kube-api-access-4xj8v\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.379550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.379551 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.379751 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.379692 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.380288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.380240 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.381906 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.381886 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.443549 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.443499 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xj8v\" (UniqueName: \"kubernetes.io/projected/1992a42f-a61e-4234-b4c0-2d1a21494785-kube-api-access-4xj8v\") pod \"maas-api-revoke-sub-f05f7811-fm22t\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.480972 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.480943 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq5br\" (UniqueName: \"kubernetes.io/projected/ef7324e8-e1af-4561-aa01-6fcee3cb4717-kube-api-access-zq5br\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.481096 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.481038 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:22.481096 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.481047 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.481096 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.481082 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.481706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.481687 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.483470 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.483452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.556669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.542777 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq5br\" (UniqueName: \"kubernetes.io/projected/ef7324e8-e1af-4561-aa01-6fcee3cb4717-kube-api-access-zq5br\") pod \"maas-api-revoke-sub-4be015e3-92hvp\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.556669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.544508 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:22.860217 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.860191 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t"] Nov 01 01:12:22.860793 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:12:22.860767 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1992a42f_a61e_4234_b4c0_2d1a21494785.slice/crio-b1556a41486ae5cf4ff22f39fbfbae56257dc63a51aa5a4a03a24f861f4505f9 WatchSource:0}: Error finding container b1556a41486ae5cf4ff22f39fbfbae56257dc63a51aa5a4a03a24f861f4505f9: Status 404 returned error can't find the container with id b1556a41486ae5cf4ff22f39fbfbae56257dc63a51aa5a4a03a24f861f4505f9 Nov 01 01:12:22.967642 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.967594 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp"] Nov 01 01:12:22.968532 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:12:22.968511 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7324e8_e1af_4561_aa01_6fcee3cb4717.slice/crio-f2b674de82f494dca49e64aa479af579b181ab24a95fce96dab75ac3f7653f62 WatchSource:0}: Error finding container f2b674de82f494dca49e64aa479af579b181ab24a95fce96dab75ac3f7653f62: Status 404 returned error can't find the container with id f2b674de82f494dca49e64aa479af579b181ab24a95fce96dab75ac3f7653f62 Nov 01 01:12:22.996690 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.996660 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" event={"ID":"1992a42f-a61e-4234-b4c0-2d1a21494785","Type":"ContainerStarted","Data":"b1556a41486ae5cf4ff22f39fbfbae56257dc63a51aa5a4a03a24f861f4505f9"} Nov 01 01:12:22.998010 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:22.997983 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" event={"ID":"ef7324e8-e1af-4561-aa01-6fcee3cb4717","Type":"ContainerStarted","Data":"f2b674de82f494dca49e64aa479af579b181ab24a95fce96dab75ac3f7653f62"} Nov 01 01:12:23.612401 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:23.612363 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" podUID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:12:24.012612 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:24.012569 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" event={"ID":"1992a42f-a61e-4234-b4c0-2d1a21494785","Type":"ContainerStarted","Data":"be5927e3215f5e65a6b7ed9d97d56543978f199299b05510cb479b65de75677a"} Nov 01 01:12:24.015646 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:24.015592 4910 generic.go:356] "Generic (PLEG): container finished" podID="ef7324e8-e1af-4561-aa01-6fcee3cb4717" containerID="1611df6415c41312a82e49623f3a36aac7d072d1c668523c7a0e800d3d4f1f05" exitCode=0 Nov 01 01:12:24.016023 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:24.015988 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" event={"ID":"ef7324e8-e1af-4561-aa01-6fcee3cb4717","Type":"ContainerDied","Data":"1611df6415c41312a82e49623f3a36aac7d072d1c668523c7a0e800d3d4f1f05"} Nov 01 01:12:24.133735 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:24.133684 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" podStartSLOduration=2.133671279 podStartE2EDuration="2.133671279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:24.130094348 +0000 UTC m=+2934.109676682" watchObservedRunningTime="2026-11-01 01:12:24.133671279 +0000 UTC m=+2934.113253611" Nov 01 01:12:25.024704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.024672 4910 generic.go:356] "Generic (PLEG): container finished" podID="1992a42f-a61e-4234-b4c0-2d1a21494785" containerID="be5927e3215f5e65a6b7ed9d97d56543978f199299b05510cb479b65de75677a" exitCode=0 Nov 01 01:12:25.024704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.024704 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" event={"ID":"1992a42f-a61e-4234-b4c0-2d1a21494785","Type":"ContainerDied","Data":"be5927e3215f5e65a6b7ed9d97d56543978f199299b05510cb479b65de75677a"} Nov 01 01:12:25.171861 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.171834 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:25.315787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.315707 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-openshift-ca\") pod \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " Nov 01 01:12:25.315787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.315774 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-xks-ca\") pod \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " Nov 01 01:12:25.316022 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.315859 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7324e8-e1af-4561-aa01-6fcee3cb4717-kube-api-access-zq5br\" (UniqueName: \"kubernetes.io/projected/ef7324e8-e1af-4561-aa01-6fcee3cb4717-kube-api-access-zq5br\") pod \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\" (UID: \"ef7324e8-e1af-4561-aa01-6fcee3cb4717\") " Nov 01 01:12:25.316078 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.316046 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-openshift-ca" pod "ef7324e8-e1af-4561-aa01-6fcee3cb4717" (UID: "ef7324e8-e1af-4561-aa01-6fcee3cb4717"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:25.316260 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.316242 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:25.318085 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.318054 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-xks-ca" pod "ef7324e8-e1af-4561-aa01-6fcee3cb4717" (UID: "ef7324e8-e1af-4561-aa01-6fcee3cb4717"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:25.318204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.318118 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7324e8-e1af-4561-aa01-6fcee3cb4717-kube-api-access-zq5br" pod "ef7324e8-e1af-4561-aa01-6fcee3cb4717" (UID: "ef7324e8-e1af-4561-aa01-6fcee3cb4717"). InnerVolumeSpecName "kube-api-access-zq5br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:25.418043 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.418005 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7324e8-e1af-4561-aa01-6fcee3cb4717-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:25.418043 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.418039 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq5br\" (UniqueName: \"kubernetes.io/projected/ef7324e8-e1af-4561-aa01-6fcee3cb4717-kube-api-access-zq5br\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:25.727674 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.727557 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr"] Nov 01 01:12:25.728151 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.728132 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ef7324e8-e1af-4561-aa01-6fcee3cb4717" containerName="revoke-keys" Nov 01 01:12:25.728262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.728155 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7324e8-e1af-4561-aa01-6fcee3cb4717" containerName="revoke-keys" Nov 01 01:12:25.728323 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.728290 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7324e8-e1af-4561-aa01-6fcee3cb4717" containerName="revoke-keys" Nov 01 01:12:25.731574 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.731553 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.789396 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.789358 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr"] Nov 01 01:12:25.822382 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.822352 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.822547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.822426 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.822547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.822489 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8cbk\" (UniqueName: \"kubernetes.io/projected/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-kube-api-access-v8cbk\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.924104 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.924067 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.924290 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.924142 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.924290 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.924168 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8cbk\" (UniqueName: \"kubernetes.io/projected/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-kube-api-access-v8cbk\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.924770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.924746 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.926811 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.926787 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:25.985665 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:25.985566 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8cbk\" (UniqueName: \"kubernetes.io/projected/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-kube-api-access-v8cbk\") pod \"maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:26.033999 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.033965 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" Nov 01 01:12:26.033999 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.033985 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp" event={"ID":"ef7324e8-e1af-4561-aa01-6fcee3cb4717","Type":"ContainerDied","Data":"f2b674de82f494dca49e64aa479af579b181ab24a95fce96dab75ac3f7653f62"} Nov 01 01:12:26.034516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.034045 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b674de82f494dca49e64aa479af579b181ab24a95fce96dab75ac3f7653f62" Nov 01 01:12:26.045548 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.045517 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:26.240095 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.240036 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:26.431789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.431759 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1992a42f-a61e-4234-b4c0-2d1a21494785-kube-api-access-4xj8v\" (UniqueName: \"kubernetes.io/projected/1992a42f-a61e-4234-b4c0-2d1a21494785-kube-api-access-4xj8v\") pod \"1992a42f-a61e-4234-b4c0-2d1a21494785\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " Nov 01 01:12:26.431956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.431891 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-openshift-ca\") pod \"1992a42f-a61e-4234-b4c0-2d1a21494785\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " Nov 01 01:12:26.431956 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.431928 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-xks-ca\") pod \"1992a42f-a61e-4234-b4c0-2d1a21494785\" (UID: \"1992a42f-a61e-4234-b4c0-2d1a21494785\") " Nov 01 01:12:26.432308 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.432276 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-openshift-ca" pod "1992a42f-a61e-4234-b4c0-2d1a21494785" (UID: "1992a42f-a61e-4234-b4c0-2d1a21494785"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:26.434090 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.434065 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-xks-ca" pod "1992a42f-a61e-4234-b4c0-2d1a21494785" (UID: "1992a42f-a61e-4234-b4c0-2d1a21494785"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:26.434226 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.434161 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1992a42f-a61e-4234-b4c0-2d1a21494785-kube-api-access-4xj8v" pod "1992a42f-a61e-4234-b4c0-2d1a21494785" (UID: "1992a42f-a61e-4234-b4c0-2d1a21494785"). InnerVolumeSpecName "kube-api-access-4xj8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:26.448936 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:12:26.448910 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdcdfed_b38a_4085_a9d8_ae5ede7e3a29.slice/crio-8063038e8df8bcdc6f1968ec01cb40d9ad81fe805b9ab099c4544372f3c23cc6 WatchSource:0}: Error finding container 8063038e8df8bcdc6f1968ec01cb40d9ad81fe805b9ab099c4544372f3c23cc6: Status 404 returned error can't find the container with id 8063038e8df8bcdc6f1968ec01cb40d9ad81fe805b9ab099c4544372f3c23cc6 Nov 01 01:12:26.450313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.450290 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr"] Nov 01 01:12:26.533725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.533697 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xj8v\" (UniqueName: \"kubernetes.io/projected/1992a42f-a61e-4234-b4c0-2d1a21494785-kube-api-access-4xj8v\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:26.533725 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.533725 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:26.533937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:26.533736 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1992a42f-a61e-4234-b4c0-2d1a21494785-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:27.042138 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:27.042051 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" Nov 01 01:12:27.042138 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:27.042090 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t" event={"ID":"1992a42f-a61e-4234-b4c0-2d1a21494785","Type":"ContainerDied","Data":"b1556a41486ae5cf4ff22f39fbfbae56257dc63a51aa5a4a03a24f861f4505f9"} Nov 01 01:12:27.042597 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:27.042151 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1556a41486ae5cf4ff22f39fbfbae56257dc63a51aa5a4a03a24f861f4505f9" Nov 01 01:12:27.043886 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:27.043862 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" event={"ID":"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29","Type":"ContainerStarted","Data":"ae3a545d101af62914bc5c6ff5213947241d4b493e81ba97b4d4e81f97fc1b60"} Nov 01 01:12:27.044009 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:27.043905 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" event={"ID":"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29","Type":"ContainerStarted","Data":"8063038e8df8bcdc6f1968ec01cb40d9ad81fe805b9ab099c4544372f3c23cc6"} Nov 01 01:12:27.161939 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:27.161889 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" podStartSLOduration=2.16187492 podStartE2EDuration="2.16187492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:27.159360263 +0000 UTC m=+2937.138942599" watchObservedRunningTime="2026-11-01 01:12:27.16187492 +0000 UTC m=+2937.141457256" Nov 01 01:12:28.054333 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.054296 4910 generic.go:356] "Generic (PLEG): container finished" podID="7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" containerID="ae3a545d101af62914bc5c6ff5213947241d4b493e81ba97b4d4e81f97fc1b60" exitCode=0 Nov 01 01:12:28.054820 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.054383 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" event={"ID":"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29","Type":"ContainerDied","Data":"ae3a545d101af62914bc5c6ff5213947241d4b493e81ba97b4d4e81f97fc1b60"} Nov 01 01:12:28.385205 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.385110 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7"] Nov 01 01:12:28.385393 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.385345 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" podUID="50f3a555-9cc3-49d3-b62d-db6c0b3519e5" containerName="discovery" containerID="cri-o://8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e" gracePeriod=30 Nov 01 01:12:28.647481 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.647428 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 01:12:28.654710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.654688 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-kubeconfig\") pod \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " Nov 01 01:12:28.654837 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.654746 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-ca-configmap\") pod \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " Nov 01 01:12:28.654837 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.654817 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-cacerts\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-cacerts\") pod \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " Nov 01 01:12:28.654959 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.654914 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-token\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-token\") pod \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " Nov 01 01:12:28.655016 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.654986 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-kube-api-access-vxdb4\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-kube-api-access-vxdb4\") pod \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " Nov 01 01:12:28.655073 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.655021 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-dns-cert\") pod \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " Nov 01 01:12:28.655073 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.655050 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-local-certs\") pod \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\" (UID: \"50f3a555-9cc3-49d3-b62d-db6c0b3519e5\") " Nov 01 01:12:28.655182 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.655152 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-ca-configmap" pod "50f3a555-9cc3-49d3-b62d-db6c0b3519e5" (UID: "50f3a555-9cc3-49d3-b62d-db6c0b3519e5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:28.655498 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.655462 4910 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-ca-configmap\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:28.657837 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.657786 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-kubeconfig" pod "50f3a555-9cc3-49d3-b62d-db6c0b3519e5" (UID: "50f3a555-9cc3-49d3-b62d-db6c0b3519e5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:28.657961 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.657886 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-dns-cert" pod "50f3a555-9cc3-49d3-b62d-db6c0b3519e5" (UID: "50f3a555-9cc3-49d3-b62d-db6c0b3519e5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:28.657961 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.657943 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-cacerts" pod "50f3a555-9cc3-49d3-b62d-db6c0b3519e5" (UID: "50f3a555-9cc3-49d3-b62d-db6c0b3519e5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:28.658253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.658222 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-token" pod "50f3a555-9cc3-49d3-b62d-db6c0b3519e5" (UID: "50f3a555-9cc3-49d3-b62d-db6c0b3519e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:28.658405 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.658386 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-kube-api-access-vxdb4" pod "50f3a555-9cc3-49d3-b62d-db6c0b3519e5" (UID: "50f3a555-9cc3-49d3-b62d-db6c0b3519e5"). InnerVolumeSpecName "kube-api-access-vxdb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:28.658570 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.658546 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-local-certs" pod "50f3a555-9cc3-49d3-b62d-db6c0b3519e5" (UID: "50f3a555-9cc3-49d3-b62d-db6c0b3519e5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:28.756149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.756112 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:28.756149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.756145 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdb4\" (UniqueName: \"kubernetes.io/projected/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-kube-api-access-vxdb4\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:28.756364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.756159 4910 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-csr-dns-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:28.756364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.756172 4910 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-local-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:28.756364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.756184 4910 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-istio-kubeconfig\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:28.756364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:28.756195 4910 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50f3a555-9cc3-49d3-b62d-db6c0b3519e5-cacerts\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:29.073446 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.073408 4910 generic.go:356] "Generic (PLEG): container finished" podID="50f3a555-9cc3-49d3-b62d-db6c0b3519e5" containerID="8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e" exitCode=0 Nov 01 01:12:29.073971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.073474 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" Nov 01 01:12:29.073971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.073490 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" event={"ID":"50f3a555-9cc3-49d3-b62d-db6c0b3519e5","Type":"ContainerDied","Data":"8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e"} Nov 01 01:12:29.073971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.073577 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7" event={"ID":"50f3a555-9cc3-49d3-b62d-db6c0b3519e5","Type":"ContainerDied","Data":"1df3c854b9b750f185a3558edea10d9e17642a1a9ff7247793a1a472809e881a"} Nov 01 01:12:29.073971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.073604 4910 scope.go:122] "RemoveContainer" containerID="8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e" Nov 01 01:12:29.089861 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.089841 4910 scope.go:122] "RemoveContainer" containerID="8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e" Nov 01 01:12:29.090173 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:12:29.090151 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e\": container with ID starting with 8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e not found: ID does not exist" containerID="8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e" Nov 01 01:12:29.090241 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.090184 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e"} err="failed to get container status \"8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e\": rpc error: code = NotFound desc = could not find container \"8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e\": container with ID starting with 8f65803fd0f5a9e6203be422d2ca5cbf1bac8bdfabd14b357544f2b8d5bede4e not found: ID does not exist" Nov 01 01:12:29.210352 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.210327 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:29.240661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.240640 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7"] Nov 01 01:12:29.291456 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.291427 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzkr7"] Nov 01 01:12:29.366851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.366771 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-openshift-ca\") pod \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " Nov 01 01:12:29.366851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.366838 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-xks-ca\") pod \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " Nov 01 01:12:29.367077 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.366875 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-kube-api-access-v8cbk\" (UniqueName: \"kubernetes.io/projected/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-kube-api-access-v8cbk\") pod \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\" (UID: \"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29\") " Nov 01 01:12:29.367225 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.367200 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-openshift-ca" pod "7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" (UID: "7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:29.369654 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.369603 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-kube-api-access-v8cbk" pod "7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" (UID: "7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29"). InnerVolumeSpecName "kube-api-access-v8cbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:29.369959 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.369937 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-xks-ca" pod "7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" (UID: "7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:29.467797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.467767 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:29.467797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.467790 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8cbk\" (UniqueName: \"kubernetes.io/projected/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-kube-api-access-v8cbk\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:29.467797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:29.467801 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:30.085746 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:30.085704 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" event={"ID":"7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29","Type":"ContainerDied","Data":"8063038e8df8bcdc6f1968ec01cb40d9ad81fe805b9ab099c4544372f3c23cc6"} Nov 01 01:12:30.085746 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:30.085759 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr" Nov 01 01:12:30.086326 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:30.085768 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8063038e8df8bcdc6f1968ec01cb40d9ad81fe805b9ab099c4544372f3c23cc6" Nov 01 01:12:30.126852 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:30.126828 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk"] Nov 01 01:12:30.177777 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:30.177741 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6"] Nov 01 01:12:30.195039 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:30.195004 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f3a555-9cc3-49d3-b62d-db6c0b3519e5" path="/var/lib/kubelet/pods/50f3a555-9cc3-49d3-b62d-db6c0b3519e5/volumes" Nov 01 01:12:32.981952 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:32.981922 4910 scope.go:122] "RemoveContainer" containerID="284bfc389881e84a18e5afad2055506367bd18312a0bfa2ea337e1f7d2cd0423" Nov 01 01:12:32.991784 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:32.991765 4910 scope.go:122] "RemoveContainer" containerID="3d03f945aaa7d4b1a5669da9713b146ce72596bf1ba3623fce31ea8cf05a0a71" Nov 01 01:12:33.050831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:33.050809 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8"] Nov 01 01:12:33.101149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:33.101125 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2484fb0e-s6ts8"] Nov 01 01:12:33.104371 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:33.104349 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn"] Nov 01 01:12:33.107148 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:33.107127 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b36c398-4hbqn"] Nov 01 01:12:33.612562 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:33.612522 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" podUID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:12:34.191294 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:34.191261 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1982ee14-47f4-4a02-9040-6b1200ca9864" path="/var/lib/kubelet/pods/1982ee14-47f4-4a02-9040-6b1200ca9864/volumes" Nov 01 01:12:34.191733 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:34.191716 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee42edb-342d-478a-a090-368586987ac9" path="/var/lib/kubelet/pods/5ee42edb-342d-478a-a090-368586987ac9/volumes" Nov 01 01:12:36.866966 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.866925 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml"] Nov 01 01:12:36.867524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867448 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1992a42f-a61e-4234-b4c0-2d1a21494785" containerName="revoke-keys" Nov 01 01:12:36.867524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867463 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1992a42f-a61e-4234-b4c0-2d1a21494785" containerName="revoke-keys" Nov 01 01:12:36.867524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867488 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" containerName="revoke-keys" Nov 01 01:12:36.867524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867493 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" containerName="revoke-keys" Nov 01 01:12:36.867524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867502 4910 cpu_manager.go:414] "containerMap: removing container" podUID="50f3a555-9cc3-49d3-b62d-db6c0b3519e5" containerName="discovery" Nov 01 01:12:36.867524 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867507 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f3a555-9cc3-49d3-b62d-db6c0b3519e5" containerName="discovery" Nov 01 01:12:36.867769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867594 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1992a42f-a61e-4234-b4c0-2d1a21494785" containerName="revoke-keys" Nov 01 01:12:36.867769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867604 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" containerName="revoke-keys" Nov 01 01:12:36.867769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.867610 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f3a555-9cc3-49d3-b62d-db6c0b3519e5" containerName="discovery" Nov 01 01:12:36.872544 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.872525 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.924793 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.924745 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7fdf2c-openshift-default-dockercfg-dk7pr\"" Nov 01 01:12:36.941962 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.941928 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942173 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.941967 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942173 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.942060 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942173 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.942095 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.942181 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809745fa-9b1c-4c29-8b67-8162926139db-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.942273 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jcw\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-kube-api-access-t8jcw\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942467 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.942348 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942467 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.942406 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:36.942467 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:36.942443 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809745fa-9b1c-4c29-8b67-8162926139db-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.043511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043477 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.043710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.043710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043553 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809745fa-9b1c-4c29-8b67-8162926139db-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.043710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043574 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.043710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043588 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.043710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043668 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.043710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043698 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.044148 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043763 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809745fa-9b1c-4c29-8b67-8162926139db-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.044148 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.043795 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jcw\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-kube-api-access-t8jcw\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.044148 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.044126 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-data\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.044328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.044158 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-certs\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.044328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.044292 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.044485 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.044429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-credential-socket\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.044537 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.044523 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809745fa-9b1c-4c29-8b67-8162926139db-istiod-ca-cert\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.046689 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.046663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-envoy\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.046887 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.046872 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809745fa-9b1c-4c29-8b67-8162926139db-istio-podinfo\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.075738 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.075711 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml"] Nov 01 01:12:37.106096 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.106059 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jcw\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-kube-api-access-t8jcw\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.108026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.107994 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-istio-token\") pod \"e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.186436 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.186366 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:37.619363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.619330 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml"] Nov 01 01:12:37.620193 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:12:37.620168 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809745fa_9b1c_4c29_8b67_8162926139db.slice/crio-d45dc63db0554215a8b22f1bb0a7d5cfdf8c8d8aed5b2eb6c28b36e1d8927365 WatchSource:0}: Error finding container d45dc63db0554215a8b22f1bb0a7d5cfdf8c8d8aed5b2eb6c28b36e1d8927365: Status 404 returned error can't find the container with id d45dc63db0554215a8b22f1bb0a7d5cfdf8c8d8aed5b2eb6c28b36e1d8927365 Nov 01 01:12:37.622467 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.622436 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:37.622549 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.622508 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:37.622549 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:37.622538 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:12:38.144937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:38.144900 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" event={"ID":"809745fa-9b1c-4c29-8b67-8162926139db","Type":"ContainerStarted","Data":"9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112"} Nov 01 01:12:38.145398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:38.144979 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" event={"ID":"809745fa-9b1c-4c29-8b67-8162926139db","Type":"ContainerStarted","Data":"d45dc63db0554215a8b22f1bb0a7d5cfdf8c8d8aed5b2eb6c28b36e1d8927365"} Nov 01 01:12:38.188045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:38.187995 4910 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 01:12:38.188220 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:38.188053 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 01:12:38.193686 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:38.193662 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:38.267320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:38.267260 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podStartSLOduration=2.267246449 podStartE2EDuration="2.267246449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:12:38.264387411 +0000 UTC m=+2948.243969746" watchObservedRunningTime="2026-11-01 01:12:38.267246449 +0000 UTC m=+2948.246828794" Nov 01 01:12:39.187116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:39.187081 4910 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 01:12:39.187665 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:39.187144 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 01:12:40.192460 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:40.192413 4910 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 01:12:40.193066 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:40.192483 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 01:12:41.187340 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.187304 4910 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 01:12:41.187557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.187388 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 01:12:41.514207 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.514181 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 01:12:41.608542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.608501 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365ddc9f-f068-42b9-8920-b4fcca9dc67c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365ddc9f-f068-42b9-8920-b4fcca9dc67c-plugins-config-volume\") pod \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " Nov 01 01:12:41.608736 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.608617 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365ddc9f-f068-42b9-8920-b4fcca9dc67c-kube-api-access-5dbx5\" (UniqueName: \"kubernetes.io/projected/365ddc9f-f068-42b9-8920-b4fcca9dc67c-kube-api-access-5dbx5\") pod \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\" (UID: \"365ddc9f-f068-42b9-8920-b4fcca9dc67c\") " Nov 01 01:12:41.608905 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.608879 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365ddc9f-f068-42b9-8920-b4fcca9dc67c-plugins-config-volume" pod "365ddc9f-f068-42b9-8920-b4fcca9dc67c" (UID: "365ddc9f-f068-42b9-8920-b4fcca9dc67c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:41.609083 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.609063 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365ddc9f-f068-42b9-8920-b4fcca9dc67c-plugins-config-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:41.611515 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.611488 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365ddc9f-f068-42b9-8920-b4fcca9dc67c-kube-api-access-5dbx5" pod "365ddc9f-f068-42b9-8920-b4fcca9dc67c" (UID: "365ddc9f-f068-42b9-8920-b4fcca9dc67c"). InnerVolumeSpecName "kube-api-access-5dbx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:41.709717 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:41.709689 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbx5\" (UniqueName: \"kubernetes.io/projected/365ddc9f-f068-42b9-8920-b4fcca9dc67c-kube-api-access-5dbx5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:42.186712 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.186672 4910 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 01:12:42.186893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.186727 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 01:12:42.205917 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.205885 4910 generic.go:356] "Generic (PLEG): container finished" podID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" containerID="73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8" exitCode=0 Nov 01 01:12:42.206050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.205936 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" Nov 01 01:12:42.206050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.205970 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" event={"ID":"365ddc9f-f068-42b9-8920-b4fcca9dc67c","Type":"ContainerDied","Data":"73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8"} Nov 01 01:12:42.206050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.206022 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp" event={"ID":"365ddc9f-f068-42b9-8920-b4fcca9dc67c","Type":"ContainerDied","Data":"a61175c963791ae0662124a596a396114d55e5768542d8c3812505eb0581ee6f"} Nov 01 01:12:42.206050 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.206044 4910 scope.go:122] "RemoveContainer" containerID="73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8" Nov 01 01:12:42.216692 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.216673 4910 scope.go:122] "RemoveContainer" containerID="73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8" Nov 01 01:12:42.216941 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:12:42.216921 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8\": container with ID starting with 73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8 not found: ID does not exist" containerID="73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8" Nov 01 01:12:42.216985 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.216949 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8"} err="failed to get container status \"73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8\": rpc error: code = NotFound desc = could not find container \"73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8\": container with ID starting with 73ad1614717543d1161b63b03aba6dff74749c7f22b1037de993c00bf85834a8 not found: ID does not exist" Nov 01 01:12:42.395785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.395742 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp"] Nov 01 01:12:42.403018 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:42.402981 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-30fb19-5b7ccf879f-tlbgp"] Nov 01 01:12:43.103478 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:12:43.103436 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:12:43.187342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:43.187307 4910 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 01 01:12:43.187536 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:43.187367 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 01 01:12:44.192149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:44.192116 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" path="/var/lib/kubelet/pods/365ddc9f-f068-42b9-8920-b4fcca9dc67c/volumes" Nov 01 01:12:44.269155 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:44.269120 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:44.269572 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:44.269552 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:44.270508 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:44.270478 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:12:45.147334 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.147287 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" containerID="cri-o://5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1" gracePeriod=45 Nov 01 01:12:45.202356 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.202168 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" containerID="cri-o://da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c" gracePeriod=45 Nov 01 01:12:45.485989 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.485957 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 01:12:45.514827 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.514803 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 01:12:45.547586 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547555 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2895da5-d473-4c4b-8288-8a055f3fb483-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2895da5-d473-4c4b-8288-8a055f3fb483-tls-certs\") pod \"f2895da5-d473-4c4b-8288-8a055f3fb483\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " Nov 01 01:12:45.547797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547600 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-home\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-home\") pod \"f2895da5-d473-4c4b-8288-8a055f3fb483\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " Nov 01 01:12:45.547797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547651 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e41946-7966-4980-8379-055ec7181838-tls-certs\" (UniqueName: \"kubernetes.io/secret/58e41946-7966-4980-8379-055ec7181838-tls-certs\") pod \"58e41946-7966-4980-8379-055ec7181838\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " Nov 01 01:12:45.547797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547688 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e41946-7966-4980-8379-055ec7181838-kube-api-access-shqfg\" (UniqueName: \"kubernetes.io/projected/58e41946-7966-4980-8379-055ec7181838-kube-api-access-shqfg\") pod \"58e41946-7966-4980-8379-055ec7181838\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " Nov 01 01:12:45.547797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547744 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2895da5-d473-4c4b-8288-8a055f3fb483-kube-api-access-m829z\" (UniqueName: \"kubernetes.io/projected/f2895da5-d473-4c4b-8288-8a055f3fb483-kube-api-access-m829z\") pod \"f2895da5-d473-4c4b-8288-8a055f3fb483\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " Nov 01 01:12:45.547797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547775 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-tmp-dir\") pod \"58e41946-7966-4980-8379-055ec7181838\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " Nov 01 01:12:45.548075 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547813 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-home\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-home\") pod \"58e41946-7966-4980-8379-055ec7181838\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " Nov 01 01:12:45.548075 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.547897 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-dshm\") pod \"58e41946-7966-4980-8379-055ec7181838\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " Nov 01 01:12:45.548262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548237 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-tmp-dir" pod "58e41946-7966-4980-8379-055ec7181838" (UID: "58e41946-7966-4980-8379-055ec7181838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.548386 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548365 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-model-cache\") pod \"58e41946-7966-4980-8379-055ec7181838\" (UID: \"58e41946-7966-4980-8379-055ec7181838\") " Nov 01 01:12:45.548474 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548417 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-tmp-dir\") pod \"f2895da5-d473-4c4b-8288-8a055f3fb483\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " Nov 01 01:12:45.548663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548608 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-home" pod "58e41946-7966-4980-8379-055ec7181838" (UID: "58e41946-7966-4980-8379-055ec7181838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.548780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548698 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-model-cache\") pod \"f2895da5-d473-4c4b-8288-8a055f3fb483\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " Nov 01 01:12:45.548780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548738 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-dshm\") pod \"f2895da5-d473-4c4b-8288-8a055f3fb483\" (UID: \"f2895da5-d473-4c4b-8288-8a055f3fb483\") " Nov 01 01:12:45.548900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548824 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-model-cache" pod "58e41946-7966-4980-8379-055ec7181838" (UID: "58e41946-7966-4980-8379-055ec7181838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.548900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.548829 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-home" pod "f2895da5-d473-4c4b-8288-8a055f3fb483" (UID: "f2895da5-d473-4c4b-8288-8a055f3fb483"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.549285 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.549056 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-tmp-dir" pod "f2895da5-d473-4c4b-8288-8a055f3fb483" (UID: "f2895da5-d473-4c4b-8288-8a055f3fb483"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.549285 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.549204 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.549285 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.549226 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.549285 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.549239 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.549285 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.549251 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.549285 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.549257 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-model-cache" pod "f2895da5-d473-4c4b-8288-8a055f3fb483" (UID: "f2895da5-d473-4c4b-8288-8a055f3fb483"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.549285 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.549267 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.550734 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.550706 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-dshm" pod "58e41946-7966-4980-8379-055ec7181838" (UID: "58e41946-7966-4980-8379-055ec7181838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.551010 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.550982 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e41946-7966-4980-8379-055ec7181838-tls-certs" pod "58e41946-7966-4980-8379-055ec7181838" (UID: "58e41946-7966-4980-8379-055ec7181838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:45.551101 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.550988 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2895da5-d473-4c4b-8288-8a055f3fb483-tls-certs" pod "f2895da5-d473-4c4b-8288-8a055f3fb483" (UID: "f2895da5-d473-4c4b-8288-8a055f3fb483"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:45.551465 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.551440 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-dshm" pod "f2895da5-d473-4c4b-8288-8a055f3fb483" (UID: "f2895da5-d473-4c4b-8288-8a055f3fb483"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:12:45.552512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.552080 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e41946-7966-4980-8379-055ec7181838-kube-api-access-shqfg" pod "58e41946-7966-4980-8379-055ec7181838" (UID: "58e41946-7966-4980-8379-055ec7181838"). InnerVolumeSpecName "kube-api-access-shqfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:45.552618 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.552585 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2895da5-d473-4c4b-8288-8a055f3fb483-kube-api-access-m829z" pod "f2895da5-d473-4c4b-8288-8a055f3fb483" (UID: "f2895da5-d473-4c4b-8288-8a055f3fb483"). InnerVolumeSpecName "kube-api-access-m829z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:45.650780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.650738 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2895da5-d473-4c4b-8288-8a055f3fb483-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.650780 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.650780 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e41946-7966-4980-8379-055ec7181838-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.651040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.650796 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqfg\" (UniqueName: \"kubernetes.io/projected/58e41946-7966-4980-8379-055ec7181838-kube-api-access-shqfg\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.651040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.650812 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m829z\" (UniqueName: \"kubernetes.io/projected/f2895da5-d473-4c4b-8288-8a055f3fb483-kube-api-access-m829z\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.651040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.650827 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e41946-7966-4980-8379-055ec7181838-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.651040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.650840 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:45.651040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:45.650854 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2895da5-d473-4c4b-8288-8a055f3fb483-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:46.241427 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.241389 4910 generic.go:356] "Generic (PLEG): container finished" podID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerID="da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c" exitCode=0 Nov 01 01:12:46.241919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.241441 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" Nov 01 01:12:46.241919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.241471 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" event={"ID":"f2895da5-d473-4c4b-8288-8a055f3fb483","Type":"ContainerDied","Data":"da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c"} Nov 01 01:12:46.241919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.241527 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" event={"ID":"f2895da5-d473-4c4b-8288-8a055f3fb483","Type":"ContainerDied","Data":"a1ff0d4042e2e3ae951f9f9c85a422e92b12ee2061e589d61548fbda76e98ba0"} Nov 01 01:12:46.241919 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.241556 4910 scope.go:122] "RemoveContainer" containerID="da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c" Nov 01 01:12:46.244605 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.244583 4910 generic.go:356] "Generic (PLEG): container finished" podID="58e41946-7966-4980-8379-055ec7181838" containerID="5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1" exitCode=0 Nov 01 01:12:46.246860 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.244652 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" event={"ID":"58e41946-7966-4980-8379-055ec7181838","Type":"ContainerDied","Data":"5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1"} Nov 01 01:12:46.247023 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.246887 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" event={"ID":"58e41946-7966-4980-8379-055ec7181838","Type":"ContainerDied","Data":"9160c3cc2efbc4ef95d4d30b9b6168cf7b903b2696b271c6e3695f3698244238"} Nov 01 01:12:46.247023 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.244767 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk" Nov 01 01:12:46.254084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.254063 4910 scope.go:122] "RemoveContainer" containerID="da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c" Nov 01 01:12:46.254410 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:12:46.254385 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c\": container with ID starting with da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c not found: ID does not exist" containerID="da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c" Nov 01 01:12:46.254493 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.254420 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c"} err="failed to get container status \"da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c\": rpc error: code = NotFound desc = could not find container \"da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c\": container with ID starting with da561f85e9a4b9ea31de9d1a301c90c622f747429b6c8393170e7998e9035b9c not found: ID does not exist" Nov 01 01:12:46.254493 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.254442 4910 scope.go:122] "RemoveContainer" containerID="5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1" Nov 01 01:12:46.264403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.264384 4910 scope.go:122] "RemoveContainer" containerID="5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1" Nov 01 01:12:46.264681 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:12:46.264658 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1\": container with ID starting with 5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1 not found: ID does not exist" containerID="5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1" Nov 01 01:12:46.264773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.264687 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1"} err="failed to get container status \"5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1\": rpc error: code = NotFound desc = could not find container \"5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1\": container with ID starting with 5b49df181e5d8902abe6c5bc585c38aea3219025de82619d70e2a51ffb7247f1 not found: ID does not exist" Nov 01 01:12:46.407397 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.407368 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk"] Nov 01 01:12:46.457039 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.456972 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/distinct-2-w6-22d7b2-kserve-8c645cd9c-lgwvk"] Nov 01 01:12:46.495910 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.495857 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 01:12:46.520186 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.520155 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6"] Nov 01 01:12:46.531947 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:46.531915 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-22d7b2/simulator-w6-22d7b2-kserve-59cccf57cd-b57j6"] Nov 01 01:12:48.193299 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:48.193265 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e41946-7966-4980-8379-055ec7181838" path="/var/lib/kubelet/pods/58e41946-7966-4980-8379-055ec7181838/volumes" Nov 01 01:12:48.193787 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:48.193770 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" path="/var/lib/kubelet/pods/f2895da5-d473-4c4b-8288-8a055f3fb483/volumes" Nov 01 01:12:49.890640 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:49.890549 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8"] Nov 01 01:12:49.940602 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:49.940573 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-30fb19-54hg8"] Nov 01 01:12:50.190666 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:50.190573 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d71d4d0-0bec-4bfc-bf90-80c2aad63028" path="/var/lib/kubelet/pods/7d71d4d0-0bec-4bfc-bf90-80c2aad63028/volumes" Nov 01 01:12:54.306718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.306688 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt"] Nov 01 01:12:54.307149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307141 4910 cpu_manager.go:414] "containerMap: removing container" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" Nov 01 01:12:54.307198 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307152 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" Nov 01 01:12:54.307198 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307168 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" Nov 01 01:12:54.307198 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307173 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" Nov 01 01:12:54.307198 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307192 4910 cpu_manager.go:414] "containerMap: removing container" podUID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" containerName="payload-pre-processing" Nov 01 01:12:54.307198 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307197 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" containerName="payload-pre-processing" Nov 01 01:12:54.307350 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307261 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="365ddc9f-f068-42b9-8920-b4fcca9dc67c" containerName="payload-pre-processing" Nov 01 01:12:54.307350 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307273 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e41946-7966-4980-8379-055ec7181838" containerName="main" Nov 01 01:12:54.307350 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.307280 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2895da5-d473-4c4b-8288-8a055f3fb483" containerName="main" Nov 01 01:12:54.311911 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.311894 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.363072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.363045 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v"] Nov 01 01:12:54.366895 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.366874 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.389849 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.389503 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt"] Nov 01 01:12:54.409853 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.409829 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:12:54.410001 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.409892 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:12:54.425134 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.425112 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v"] Nov 01 01:12:54.441829 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.441800 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.441947 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.441845 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.441990 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.441957 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rn5\" (UniqueName: \"kubernetes.io/projected/0271ed6d-f127-4a5b-a01c-b909c4ada584-kube-api-access-p5rn5\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.442024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.441986 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hl6\" (UniqueName: \"kubernetes.io/projected/e3ae261e-6b6e-4c5b-b72a-61732e457371-kube-api-access-k4hl6\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.442024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.442006 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.442163 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.442145 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.542864 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.542833 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.543034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.542890 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.543034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.542929 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.543034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.543004 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rn5\" (UniqueName: \"kubernetes.io/projected/0271ed6d-f127-4a5b-a01c-b909c4ada584-kube-api-access-p5rn5\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.543177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.543034 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hl6\" (UniqueName: \"kubernetes.io/projected/e3ae261e-6b6e-4c5b-b72a-61732e457371-kube-api-access-k4hl6\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.543177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.543061 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.543681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.543611 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.543681 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.543657 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.545464 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.545435 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.545569 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.545521 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.604395 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.604332 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rn5\" (UniqueName: \"kubernetes.io/projected/0271ed6d-f127-4a5b-a01c-b909c4ada584-kube-api-access-p5rn5\") pod \"maas-api-revoke-sub-cfd0adb0-mbd2v\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:54.604520 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.604452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hl6\" (UniqueName: \"kubernetes.io/projected/e3ae261e-6b6e-4c5b-b72a-61732e457371-kube-api-access-k4hl6\") pod \"maas-api-revoke-sub-049c6ebd-7zkkt\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.625313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.625292 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:54.701332 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:54.696888 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:55.007003 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:55.006975 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt"] Nov 01 01:12:55.007935 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:12:55.007910 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ae261e_6b6e_4c5b_b72a_61732e457371.slice/crio-ef7438e8e120513d151553af737de538b01ed7516795dc2e1dcf33b66911a9dd WatchSource:0}: Error finding container ef7438e8e120513d151553af737de538b01ed7516795dc2e1dcf33b66911a9dd: Status 404 returned error can't find the container with id ef7438e8e120513d151553af737de538b01ed7516795dc2e1dcf33b66911a9dd Nov 01 01:12:55.108296 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:55.108270 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v"] Nov 01 01:12:55.109188 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:12:55.109159 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0271ed6d_f127_4a5b_a01c_b909c4ada584.slice/crio-bba59d2ed6e8e33aac6d533411f3f28f94bcbd505dcd042b577f7d07b87fb1fc WatchSource:0}: Error finding container bba59d2ed6e8e33aac6d533411f3f28f94bcbd505dcd042b577f7d07b87fb1fc: Status 404 returned error can't find the container with id bba59d2ed6e8e33aac6d533411f3f28f94bcbd505dcd042b577f7d07b87fb1fc Nov 01 01:12:55.315785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:55.315752 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" event={"ID":"e3ae261e-6b6e-4c5b-b72a-61732e457371","Type":"ContainerStarted","Data":"ef7438e8e120513d151553af737de538b01ed7516795dc2e1dcf33b66911a9dd"} Nov 01 01:12:55.321180 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:55.321147 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" event={"ID":"0271ed6d-f127-4a5b-a01c-b909c4ada584","Type":"ContainerStarted","Data":"bba59d2ed6e8e33aac6d533411f3f28f94bcbd505dcd042b577f7d07b87fb1fc"} Nov 01 01:12:56.329516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:56.329434 4910 generic.go:356] "Generic (PLEG): container finished" podID="0271ed6d-f127-4a5b-a01c-b909c4ada584" containerID="e36afdb51ff92514825d1f368f61af9ba8df9bbb52803f41cf5ec1f317aed4d3" exitCode=0 Nov 01 01:12:56.329949 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:56.329524 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" event={"ID":"0271ed6d-f127-4a5b-a01c-b909c4ada584","Type":"ContainerDied","Data":"e36afdb51ff92514825d1f368f61af9ba8df9bbb52803f41cf5ec1f317aed4d3"} Nov 01 01:12:56.331257 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:56.331235 4910 generic.go:356] "Generic (PLEG): container finished" podID="e3ae261e-6b6e-4c5b-b72a-61732e457371" containerID="7d3f7afa1d87a98d225860053131799dcd6758f8d740190669544ce5c6f974a9" exitCode=0 Nov 01 01:12:56.331372 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:56.331315 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" event={"ID":"e3ae261e-6b6e-4c5b-b72a-61732e457371","Type":"ContainerDied","Data":"7d3f7afa1d87a98d225860053131799dcd6758f8d740190669544ce5c6f974a9"} Nov 01 01:12:57.484006 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.483985 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:57.515147 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.515123 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:57.574740 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.574708 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-xks-ca\") pod \"0271ed6d-f127-4a5b-a01c-b909c4ada584\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " Nov 01 01:12:57.574912 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.574764 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-openshift-ca\") pod \"e3ae261e-6b6e-4c5b-b72a-61732e457371\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " Nov 01 01:12:57.574912 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.574797 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0271ed6d-f127-4a5b-a01c-b909c4ada584-kube-api-access-p5rn5\" (UniqueName: \"kubernetes.io/projected/0271ed6d-f127-4a5b-a01c-b909c4ada584-kube-api-access-p5rn5\") pod \"0271ed6d-f127-4a5b-a01c-b909c4ada584\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " Nov 01 01:12:57.575028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.574908 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-xks-ca\") pod \"e3ae261e-6b6e-4c5b-b72a-61732e457371\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " Nov 01 01:12:57.575028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.575016 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-openshift-ca\") pod \"0271ed6d-f127-4a5b-a01c-b909c4ada584\" (UID: \"0271ed6d-f127-4a5b-a01c-b909c4ada584\") " Nov 01 01:12:57.575151 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.575065 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ae261e-6b6e-4c5b-b72a-61732e457371-kube-api-access-k4hl6\" (UniqueName: \"kubernetes.io/projected/e3ae261e-6b6e-4c5b-b72a-61732e457371-kube-api-access-k4hl6\") pod \"e3ae261e-6b6e-4c5b-b72a-61732e457371\" (UID: \"e3ae261e-6b6e-4c5b-b72a-61732e457371\") " Nov 01 01:12:57.575226 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.575178 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-openshift-ca" pod "e3ae261e-6b6e-4c5b-b72a-61732e457371" (UID: "e3ae261e-6b6e-4c5b-b72a-61732e457371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:57.575422 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.575398 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-openshift-ca" pod "0271ed6d-f127-4a5b-a01c-b909c4ada584" (UID: "0271ed6d-f127-4a5b-a01c-b909c4ada584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:12:57.575504 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.575427 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:57.577092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.577062 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-xks-ca" pod "e3ae261e-6b6e-4c5b-b72a-61732e457371" (UID: "e3ae261e-6b6e-4c5b-b72a-61732e457371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:57.577207 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.577127 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0271ed6d-f127-4a5b-a01c-b909c4ada584-kube-api-access-p5rn5" pod "0271ed6d-f127-4a5b-a01c-b909c4ada584" (UID: "0271ed6d-f127-4a5b-a01c-b909c4ada584"). InnerVolumeSpecName "kube-api-access-p5rn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:57.577207 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.577185 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-xks-ca" pod "0271ed6d-f127-4a5b-a01c-b909c4ada584" (UID: "0271ed6d-f127-4a5b-a01c-b909c4ada584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:12:57.577400 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.577380 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ae261e-6b6e-4c5b-b72a-61732e457371-kube-api-access-k4hl6" pod "e3ae261e-6b6e-4c5b-b72a-61732e457371" (UID: "e3ae261e-6b6e-4c5b-b72a-61732e457371"). InnerVolumeSpecName "kube-api-access-k4hl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:12:57.676728 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.676485 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rn5\" (UniqueName: \"kubernetes.io/projected/0271ed6d-f127-4a5b-a01c-b909c4ada584-kube-api-access-p5rn5\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:57.676728 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.676522 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ae261e-6b6e-4c5b-b72a-61732e457371-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:57.676728 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.676539 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:57.676728 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.676552 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4hl6\" (UniqueName: \"kubernetes.io/projected/e3ae261e-6b6e-4c5b-b72a-61732e457371-kube-api-access-k4hl6\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:57.676728 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:57.676565 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0271ed6d-f127-4a5b-a01c-b909c4ada584-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:12:58.347771 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:58.347739 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" Nov 01 01:12:58.347940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:58.347771 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt" event={"ID":"e3ae261e-6b6e-4c5b-b72a-61732e457371","Type":"ContainerDied","Data":"ef7438e8e120513d151553af737de538b01ed7516795dc2e1dcf33b66911a9dd"} Nov 01 01:12:58.347940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:58.347828 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef7438e8e120513d151553af737de538b01ed7516795dc2e1dcf33b66911a9dd" Nov 01 01:12:58.349325 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:58.349298 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" event={"ID":"0271ed6d-f127-4a5b-a01c-b909c4ada584","Type":"ContainerDied","Data":"bba59d2ed6e8e33aac6d533411f3f28f94bcbd505dcd042b577f7d07b87fb1fc"} Nov 01 01:12:58.349455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:58.349330 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v" Nov 01 01:12:58.349455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:12:58.349340 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bba59d2ed6e8e33aac6d533411f3f28f94bcbd505dcd042b577f7d07b87fb1fc" Nov 01 01:13:13.050328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:13.050284 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z"] Nov 01 01:13:13.101078 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:13.101044 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31a639c6-stm7z"] Nov 01 01:13:13.103654 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:13:13.103608 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:13:14.055680 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:14.055606 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km"] Nov 01 01:13:14.102301 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:14.102263 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bcf8c35-fd7km"] Nov 01 01:13:14.190889 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:14.190856 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457aa14d-cf8a-439f-8048-1f50d4431442" path="/var/lib/kubelet/pods/457aa14d-cf8a-439f-8048-1f50d4431442/volumes" Nov 01 01:13:14.191281 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:14.191267 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb2adde-1af6-4023-962c-1f5376d964f5" path="/var/lib/kubelet/pods/7cb2adde-1af6-4023-962c-1f5376d964f5/volumes" Nov 01 01:13:21.302241 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:21.302198 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr"] Nov 01 01:13:21.351483 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:21.351452 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-22d7b2-f29gr"] Nov 01 01:13:22.191813 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:22.191778 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29" path="/var/lib/kubelet/pods/7fdcdfed-b38a-4085-a9d8-ae5ede7e3a29/volumes" Nov 01 01:13:28.390742 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:28.390705 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc"] Nov 01 01:13:28.391247 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:28.390954 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" containerID="cri-o://a7203722ce13cde00c55bd2dbebe9079b5e4b4b2aeee2bd7a8e9422a0738a191" gracePeriod=30 Nov 01 01:13:28.431759 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:28.431724 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:13:28.431894 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:28.431800 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:13:30.908690 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:30.908663 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:13:30.923269 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:30.923245 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:13:30.924015 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:30.923998 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:13:30.935163 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:30.935145 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:13:33.035184 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.035154 4910 scope.go:122] "RemoveContainer" containerID="886ada4b8f2424d6f05a264e2846639e8133e86ecd90a704d4f93cc69526bd1e" Nov 01 01:13:33.045042 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.045018 4910 scope.go:122] "RemoveContainer" containerID="df249ab79aca003715b4ef9bdbf196e5f01e1ad9e76a9b2720357d9d3e36486e" Nov 01 01:13:33.054239 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.054223 4910 scope.go:122] "RemoveContainer" containerID="b2610b89e48e7673f2ab6939f6e378126750fa9ef3a0672b35f488613dd84f1c" Nov 01 01:13:33.063013 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.062990 4910 scope.go:122] "RemoveContainer" containerID="fa28c15a18b46c2c721fea2aeb6cf8fb4c7b32e57869c772165ff445ee69a769" Nov 01 01:13:33.071785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.071763 4910 scope.go:122] "RemoveContainer" containerID="40f236470a8e64cbdd7a2c9bb984d996f8ec57d451eb613c4ae87291435b469c" Nov 01 01:13:33.080546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.080524 4910 scope.go:122] "RemoveContainer" containerID="1294aea4e75ca22b74134ba2bb7799db4af0ef8338aa4b8a4dcc32fc7754601e" Nov 01 01:13:33.089211 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.089195 4910 scope.go:122] "RemoveContainer" containerID="6eb85a902ed06f3e5aee740ad870538351e137a70da397e448a44e62e35c0a34" Nov 01 01:13:33.614212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.614173 4910 generic.go:356] "Generic (PLEG): container finished" podID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerID="a7203722ce13cde00c55bd2dbebe9079b5e4b4b2aeee2bd7a8e9422a0738a191" exitCode=0 Nov 01 01:13:33.614398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.614295 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" event={"ID":"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4","Type":"ContainerDied","Data":"a7203722ce13cde00c55bd2dbebe9079b5e4b4b2aeee2bd7a8e9422a0738a191"} Nov 01 01:13:33.656061 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.656009 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 01:13:33.739718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.739688 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-credential-socket\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.739896 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.739751 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-socket\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.739896 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.739784 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-envoy\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.739896 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.739837 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-podinfo\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.740105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.739909 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-kube-api-access-tzfsq\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-kube-api-access-tzfsq\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.740105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.739991 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-token\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-token\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.740105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740024 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-socket" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:33.740105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740074 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-credential-socket" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:33.740328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740106 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istiod-ca-cert\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.740328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740176 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-certs\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.740328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740216 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-data\") pod \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\" (UID: \"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4\") " Nov 01 01:13:33.740844 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740584 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.740844 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740607 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.740844 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740785 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-certs" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:33.741313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740873 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-data" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:33.741313 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.740953 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istiod-ca-cert" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:13:33.742746 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.742712 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-envoy" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:13:33.743058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.743030 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-token" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:13:33.743157 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.743138 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-podinfo" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:13:33.743305 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.743284 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-kube-api-access-tzfsq" pod "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" (UID: "f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4"). InnerVolumeSpecName "kube-api-access-tzfsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:13:33.842293 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.842267 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.842293 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.842291 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.842476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.842300 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfsq\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-kube-api-access-tzfsq\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.842476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.842310 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.842476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.842319 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.842476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.842329 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:33.842476 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:33.842337 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:13:34.621862 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:34.621832 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" Nov 01 01:13:34.621862 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:34.621854 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc" event={"ID":"f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4","Type":"ContainerDied","Data":"474cef4f3481942d9bb86fe80a7d4d49789a3bd793bd88ebd5d95f431ae1ef0f"} Nov 01 01:13:34.622376 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:34.621915 4910 scope.go:122] "RemoveContainer" containerID="a7203722ce13cde00c55bd2dbebe9079b5e4b4b2aeee2bd7a8e9422a0738a191" Nov 01 01:13:34.839404 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:34.839371 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc"] Nov 01 01:13:34.894977 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:34.894884 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-22d7b2-openshift-default-6c5985f547-f8pkc"] Nov 01 01:13:36.191444 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:13:36.191412 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" path="/var/lib/kubelet/pods/f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4/volumes" Nov 01 01:13:43.103092 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:13:43.103004 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:14:09.656377 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.656340 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw"] Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.656999 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0271ed6d-f127-4a5b-a01c-b909c4ada584" containerName="revoke-keys" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657017 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0271ed6d-f127-4a5b-a01c-b909c4ada584" containerName="revoke-keys" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657060 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657066 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657083 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e3ae261e-6b6e-4c5b-b72a-61732e457371" containerName="revoke-keys" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657088 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ae261e-6b6e-4c5b-b72a-61732e457371" containerName="revoke-keys" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657156 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20b65f7-9b42-4b6f-8a59-5eb5a0d13af4" containerName="istio-proxy" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657166 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ae261e-6b6e-4c5b-b72a-61732e457371" containerName="revoke-keys" Nov 01 01:14:09.658434 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.657174 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0271ed6d-f127-4a5b-a01c-b909c4ada584" containerName="revoke-keys" Nov 01 01:14:09.660225 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.660205 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.693604 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.693572 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frh7d\" (UniqueName: \"kubernetes.io/projected/52171400-83a4-4fe2-b2dc-34504dbc3259-kube-api-access-frh7d\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.693765 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.693689 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-tmp-dir\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.693765 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.693716 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-model-cache\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.693914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.693818 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-dshm\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.693914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.693848 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-home\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.693914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.693884 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52171400-83a4-4fe2-b2dc-34504dbc3259-tls-certs\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.765125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.765091 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-48d289\"/\"default-dockercfg-njvbh\"" Nov 01 01:14:09.765324 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.765153 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-48d289\"/\"test-model-48d289-kserve-self-signed-certs\"" Nov 01 01:14:09.765942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.765919 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-48d289\"/\"openshift-service-ca.crt\"" Nov 01 01:14:09.766045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.765961 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-48d289\"/\"kube-root-ca.crt\"" Nov 01 01:14:09.768576 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.768553 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw"] Nov 01 01:14:09.794770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.794746 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-home\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.794882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.794797 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52171400-83a4-4fe2-b2dc-34504dbc3259-tls-certs\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.794882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.794828 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frh7d\" (UniqueName: \"kubernetes.io/projected/52171400-83a4-4fe2-b2dc-34504dbc3259-kube-api-access-frh7d\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.794882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.794873 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-tmp-dir\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.795076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.794929 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-model-cache\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.795076 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.795037 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-dshm\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.795189 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.795162 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-home\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.795268 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.795241 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-tmp-dir\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.795389 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.795373 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-model-cache\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.797402 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.797377 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-dshm\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.797749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.797726 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52171400-83a4-4fe2-b2dc-34504dbc3259-tls-certs\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.855939 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.855909 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frh7d\" (UniqueName: \"kubernetes.io/projected/52171400-83a4-4fe2-b2dc-34504dbc3259-kube-api-access-frh7d\") pod \"test-model-48d289-kserve-647c8c6d45-rzlrw\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:09.972197 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:09.972109 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:10.358744 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:10.358583 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw"] Nov 01 01:14:10.362510 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:14:10.362457 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52171400_83a4_4fe2_b2dc_34504dbc3259.slice/crio-dd65306df8d3f4c48c8cc6dceb8281b5387a56de578be8de164cfcc8a766da55 WatchSource:0}: Error finding container dd65306df8d3f4c48c8cc6dceb8281b5387a56de578be8de164cfcc8a766da55: Status 404 returned error can't find the container with id dd65306df8d3f4c48c8cc6dceb8281b5387a56de578be8de164cfcc8a766da55 Nov 01 01:14:10.876662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:10.876608 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" event={"ID":"52171400-83a4-4fe2-b2dc-34504dbc3259","Type":"ContainerStarted","Data":"c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c"} Nov 01 01:14:10.877115 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:10.876673 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" event={"ID":"52171400-83a4-4fe2-b2dc-34504dbc3259","Type":"ContainerStarted","Data":"dd65306df8d3f4c48c8cc6dceb8281b5387a56de578be8de164cfcc8a766da55"} Nov 01 01:14:10.972616 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:10.972584 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:10.997239 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:10.997187 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" podStartSLOduration=1.997173335 podStartE2EDuration="1.997173335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:14:10.994497595 +0000 UTC m=+3040.974079929" watchObservedRunningTime="2026-11-01 01:14:10.997173335 +0000 UTC m=+3040.976755670" Nov 01 01:14:13.104031 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:14:13.103994 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:14:19.972662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:19.972616 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:19.987155 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:19.987124 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:20.957940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:20.957910 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:14:33.156751 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:33.156718 4910 scope.go:122] "RemoveContainer" containerID="81180813f2969fd98049d6b79ce2b72700654f1dc989d46a3a1d76da84132ac9" Nov 01 01:14:33.169856 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:33.169830 4910 scope.go:122] "RemoveContainer" containerID="841a8aebb96ddfe51ae86032fd339d16cec3f2396bf36738d6bbddcb4f25e5f0" Nov 01 01:14:33.179393 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:14:33.179374 4910 scope.go:122] "RemoveContainer" containerID="c5d9efc6cd6ab2973513396329537e711596856461b4a790869fdc3a0689239f" Nov 01 01:14:43.102747 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:14:43.102704 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:15:00.164341 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.164310 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk"] Nov 01 01:15:00.168354 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.168330 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4"] Nov 01 01:15:00.168519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.168467 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.172084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.172061 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.182085 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.182061 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afdb75f-def1-43d8-811e-3fbfa7afa195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.182216 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.182094 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8306c9fc-6af9-450f-9c70-47b912bc34a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.182291 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.182229 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6ml\" (UniqueName: \"kubernetes.io/projected/5afdb75f-def1-43d8-811e-3fbfa7afa195-kube-api-access-kn6ml\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.182392 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.182375 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92kpj\" (UniqueName: \"kubernetes.io/projected/8306c9fc-6af9-450f-9c70-47b912bc34a1-kube-api-access-92kpj\") pod \"maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.222865 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.222831 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:15:00.280522 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.280480 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk"] Nov 01 01:15:00.280747 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.280598 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4"] Nov 01 01:15:00.283186 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.283161 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6ml\" (UniqueName: \"kubernetes.io/projected/5afdb75f-def1-43d8-811e-3fbfa7afa195-kube-api-access-kn6ml\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.283303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.283242 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92kpj\" (UniqueName: \"kubernetes.io/projected/8306c9fc-6af9-450f-9c70-47b912bc34a1-kube-api-access-92kpj\") pod \"maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.283303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.283273 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afdb75f-def1-43d8-811e-3fbfa7afa195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.283303 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.283290 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8306c9fc-6af9-450f-9c70-47b912bc34a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.283906 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.283888 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8306c9fc-6af9-450f-9c70-47b912bc34a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.284733 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.284708 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afdb75f-def1-43d8-811e-3fbfa7afa195-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.346035 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.346003 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92kpj\" (UniqueName: \"kubernetes.io/projected/8306c9fc-6af9-450f-9c70-47b912bc34a1-kube-api-access-92kpj\") pod \"maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.346192 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.346104 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6ml\" (UniqueName: \"kubernetes.io/projected/5afdb75f-def1-43d8-811e-3fbfa7afa195-kube-api-access-kn6ml\") pod \"maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.486441 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.486357 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:00.496695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.496669 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:00.868420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.868392 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk"] Nov 01 01:15:00.869296 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:15:00.869269 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8306c9fc_6af9_450f_9c70_47b912bc34a1.slice/crio-ea5f085c08b770bdd3ef739b508d895b3d1230d9de795e9a30f7af9d8e630299 WatchSource:0}: Error finding container ea5f085c08b770bdd3ef739b508d895b3d1230d9de795e9a30f7af9d8e630299: Status 404 returned error can't find the container with id ea5f085c08b770bdd3ef739b508d895b3d1230d9de795e9a30f7af9d8e630299 Nov 01 01:15:00.920106 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:00.920086 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4"] Nov 01 01:15:00.920693 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:15:00.920670 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afdb75f_def1_43d8_811e_3fbfa7afa195.slice/crio-50097bcf262d3441d70ffa343c43597494d8a243427c4d5b22e0be02d1675cfc WatchSource:0}: Error finding container 50097bcf262d3441d70ffa343c43597494d8a243427c4d5b22e0be02d1675cfc: Status 404 returned error can't find the container with id 50097bcf262d3441d70ffa343c43597494d8a243427c4d5b22e0be02d1675cfc Nov 01 01:15:01.227381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:01.227336 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" event={"ID":"8306c9fc-6af9-450f-9c70-47b912bc34a1","Type":"ContainerStarted","Data":"517e86697ceaa97950c61c06f6ce489e4bef68e1b7fbd13ef58871c7405f3919"} Nov 01 01:15:01.227849 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:01.227389 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" event={"ID":"8306c9fc-6af9-450f-9c70-47b912bc34a1","Type":"ContainerStarted","Data":"ea5f085c08b770bdd3ef739b508d895b3d1230d9de795e9a30f7af9d8e630299"} Nov 01 01:15:01.228739 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:01.228714 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" event={"ID":"5afdb75f-def1-43d8-811e-3fbfa7afa195","Type":"ContainerStarted","Data":"50097bcf262d3441d70ffa343c43597494d8a243427c4d5b22e0be02d1675cfc"} Nov 01 01:15:01.344817 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:01.344760 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" podStartSLOduration=1.344747537 podStartE2EDuration="1.344747537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:15:01.344281607 +0000 UTC m=+3091.323863943" watchObservedRunningTime="2026-11-01 01:15:01.344747537 +0000 UTC m=+3091.324329886" Nov 01 01:15:02.237347 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:02.237259 4910 generic.go:356] "Generic (PLEG): container finished" podID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerID="517e86697ceaa97950c61c06f6ce489e4bef68e1b7fbd13ef58871c7405f3919" exitCode=60 Nov 01 01:15:02.237840 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:02.237328 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" event={"ID":"8306c9fc-6af9-450f-9c70-47b912bc34a1","Type":"ContainerDied","Data":"517e86697ceaa97950c61c06f6ce489e4bef68e1b7fbd13ef58871c7405f3919"} Nov 01 01:15:02.237840 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:02.237600 4910 scope.go:122] "RemoveContainer" containerID="517e86697ceaa97950c61c06f6ce489e4bef68e1b7fbd13ef58871c7405f3919" Nov 01 01:15:02.239353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:02.239330 4910 generic.go:356] "Generic (PLEG): container finished" podID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerID="09083a7843ec659daaeb445e42311694ec8f57878397f44ca48bf3715a4f2428" exitCode=60 Nov 01 01:15:02.239464 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:02.239391 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" event={"ID":"5afdb75f-def1-43d8-811e-3fbfa7afa195","Type":"ContainerDied","Data":"09083a7843ec659daaeb445e42311694ec8f57878397f44ca48bf3715a4f2428"} Nov 01 01:15:02.239605 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:02.239590 4910 scope.go:122] "RemoveContainer" containerID="09083a7843ec659daaeb445e42311694ec8f57878397f44ca48bf3715a4f2428" Nov 01 01:15:03.248130 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:03.248100 4910 generic.go:356] "Generic (PLEG): container finished" podID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerID="0a0f44d013d294a3366e7e1576fa01ea3fbae1b7ebe5ab3983ad5f2dcddc3a6e" exitCode=60 Nov 01 01:15:03.248594 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:03.248172 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" event={"ID":"5afdb75f-def1-43d8-811e-3fbfa7afa195","Type":"ContainerDied","Data":"0a0f44d013d294a3366e7e1576fa01ea3fbae1b7ebe5ab3983ad5f2dcddc3a6e"} Nov 01 01:15:03.248594 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:03.248234 4910 scope.go:122] "RemoveContainer" containerID="09083a7843ec659daaeb445e42311694ec8f57878397f44ca48bf3715a4f2428" Nov 01 01:15:03.248594 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:03.248481 4910 scope.go:122] "RemoveContainer" containerID="0a0f44d013d294a3366e7e1576fa01ea3fbae1b7ebe5ab3983ad5f2dcddc3a6e" Nov 01 01:15:03.248816 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:03.248787 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4_odh-ai-gateway-infra(5afdb75f-def1-43d8-811e-3fbfa7afa195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" Nov 01 01:15:03.250839 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:03.250817 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" event={"ID":"8306c9fc-6af9-450f-9c70-47b912bc34a1","Type":"ContainerStarted","Data":"172a899e1dffe201628f8acfd458689260dfa5ad0ad5614714076faa84b82cd1"} Nov 01 01:15:04.259733 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:04.259701 4910 scope.go:122] "RemoveContainer" containerID="0a0f44d013d294a3366e7e1576fa01ea3fbae1b7ebe5ab3983ad5f2dcddc3a6e" Nov 01 01:15:04.260268 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:04.259985 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4_odh-ai-gateway-infra(5afdb75f-def1-43d8-811e-3fbfa7afa195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" Nov 01 01:15:04.261515 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:04.261486 4910 generic.go:356] "Generic (PLEG): container finished" podID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerID="172a899e1dffe201628f8acfd458689260dfa5ad0ad5614714076faa84b82cd1" exitCode=60 Nov 01 01:15:04.261682 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:04.261570 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" event={"ID":"8306c9fc-6af9-450f-9c70-47b912bc34a1","Type":"ContainerDied","Data":"172a899e1dffe201628f8acfd458689260dfa5ad0ad5614714076faa84b82cd1"} Nov 01 01:15:04.261682 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:04.261654 4910 scope.go:122] "RemoveContainer" containerID="517e86697ceaa97950c61c06f6ce489e4bef68e1b7fbd13ef58871c7405f3919" Nov 01 01:15:04.261782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:04.261762 4910 scope.go:122] "RemoveContainer" containerID="172a899e1dffe201628f8acfd458689260dfa5ad0ad5614714076faa84b82cd1" Nov 01 01:15:04.262032 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:04.262014 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk_odh-ai-gateway-infra(8306c9fc-6af9-450f-9c70-47b912bc34a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" Nov 01 01:15:05.270647 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:05.270601 4910 scope.go:122] "RemoveContainer" containerID="172a899e1dffe201628f8acfd458689260dfa5ad0ad5614714076faa84b82cd1" Nov 01 01:15:05.271063 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:05.270844 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk_odh-ai-gateway-infra(8306c9fc-6af9-450f-9c70-47b912bc34a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" Nov 01 01:15:13.104171 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:13.104135 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:15:15.185578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:15.185538 4910 scope.go:122] "RemoveContainer" containerID="0a0f44d013d294a3366e7e1576fa01ea3fbae1b7ebe5ab3983ad5f2dcddc3a6e" Nov 01 01:15:16.349567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:16.349529 4910 generic.go:356] "Generic (PLEG): container finished" podID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerID="681f6db1c671973597b228227a2287f2dd17124ee8d4b147617a160c55cf6541" exitCode=60 Nov 01 01:15:16.350041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:16.349602 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" event={"ID":"5afdb75f-def1-43d8-811e-3fbfa7afa195","Type":"ContainerDied","Data":"681f6db1c671973597b228227a2287f2dd17124ee8d4b147617a160c55cf6541"} Nov 01 01:15:16.350041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:16.349680 4910 scope.go:122] "RemoveContainer" containerID="0a0f44d013d294a3366e7e1576fa01ea3fbae1b7ebe5ab3983ad5f2dcddc3a6e" Nov 01 01:15:16.350127 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:16.350067 4910 scope.go:122] "RemoveContainer" containerID="681f6db1c671973597b228227a2287f2dd17124ee8d4b147617a160c55cf6541" Nov 01 01:15:16.350365 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:16.350341 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4_odh-ai-gateway-infra(5afdb75f-def1-43d8-811e-3fbfa7afa195)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" Nov 01 01:15:17.185027 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.184993 4910 scope.go:122] "RemoveContainer" containerID="172a899e1dffe201628f8acfd458689260dfa5ad0ad5614714076faa84b82cd1" Nov 01 01:15:17.477051 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.477025 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4"] Nov 01 01:15:17.627370 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.627346 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:17.756137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.756100 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afdb75f-def1-43d8-811e-3fbfa7afa195-kube-api-access-kn6ml\" (UniqueName: \"kubernetes.io/projected/5afdb75f-def1-43d8-811e-3fbfa7afa195-kube-api-access-kn6ml\") pod \"5afdb75f-def1-43d8-811e-3fbfa7afa195\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " Nov 01 01:15:17.756137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.756138 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5afdb75f-def1-43d8-811e-3fbfa7afa195-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afdb75f-def1-43d8-811e-3fbfa7afa195-maas-api-service-ca\") pod \"5afdb75f-def1-43d8-811e-3fbfa7afa195\" (UID: \"5afdb75f-def1-43d8-811e-3fbfa7afa195\") " Nov 01 01:15:17.756534 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.756505 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afdb75f-def1-43d8-811e-3fbfa7afa195-maas-api-service-ca" pod "5afdb75f-def1-43d8-811e-3fbfa7afa195" (UID: "5afdb75f-def1-43d8-811e-3fbfa7afa195"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:15:17.758391 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.758365 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afdb75f-def1-43d8-811e-3fbfa7afa195-kube-api-access-kn6ml" pod "5afdb75f-def1-43d8-811e-3fbfa7afa195" (UID: "5afdb75f-def1-43d8-811e-3fbfa7afa195"). InnerVolumeSpecName "kube-api-access-kn6ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:15:17.857791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.857751 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6ml\" (UniqueName: \"kubernetes.io/projected/5afdb75f-def1-43d8-811e-3fbfa7afa195-kube-api-access-kn6ml\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:15:17.857791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:17.857783 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5afdb75f-def1-43d8-811e-3fbfa7afa195-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:15:18.367028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.366934 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" Nov 01 01:15:18.367028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.366975 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4" event={"ID":"5afdb75f-def1-43d8-811e-3fbfa7afa195","Type":"ContainerDied","Data":"50097bcf262d3441d70ffa343c43597494d8a243427c4d5b22e0be02d1675cfc"} Nov 01 01:15:18.367287 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.367040 4910 scope.go:122] "RemoveContainer" containerID="681f6db1c671973597b228227a2287f2dd17124ee8d4b147617a160c55cf6541" Nov 01 01:15:18.369301 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.369279 4910 generic.go:356] "Generic (PLEG): container finished" podID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerID="39034b9589c622a6e9e6e196b718cbcb19f672b5ad82c71a430ee5080e19d403" exitCode=60 Nov 01 01:15:18.369427 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.369308 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" event={"ID":"8306c9fc-6af9-450f-9c70-47b912bc34a1","Type":"ContainerDied","Data":"39034b9589c622a6e9e6e196b718cbcb19f672b5ad82c71a430ee5080e19d403"} Nov 01 01:15:18.369683 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.369660 4910 scope.go:122] "RemoveContainer" containerID="39034b9589c622a6e9e6e196b718cbcb19f672b5ad82c71a430ee5080e19d403" Nov 01 01:15:18.369920 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:18.369903 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk_odh-ai-gateway-infra(8306c9fc-6af9-450f-9c70-47b912bc34a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" Nov 01 01:15:18.380195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.380171 4910 scope.go:122] "RemoveContainer" containerID="172a899e1dffe201628f8acfd458689260dfa5ad0ad5614714076faa84b82cd1" Nov 01 01:15:18.652109 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.652031 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4"] Nov 01 01:15:18.656418 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:18.656392 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-43a0cf-29891595-hkhn4"] Nov 01 01:15:19.550773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.550739 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk"] Nov 01 01:15:19.690049 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.690021 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:19.775923 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.775830 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8306c9fc-6af9-450f-9c70-47b912bc34a1-kube-api-access-92kpj\" (UniqueName: \"kubernetes.io/projected/8306c9fc-6af9-450f-9c70-47b912bc34a1-kube-api-access-92kpj\") pod \"8306c9fc-6af9-450f-9c70-47b912bc34a1\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " Nov 01 01:15:19.775923 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.775881 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8306c9fc-6af9-450f-9c70-47b912bc34a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8306c9fc-6af9-450f-9c70-47b912bc34a1-maas-api-service-ca\") pod \"8306c9fc-6af9-450f-9c70-47b912bc34a1\" (UID: \"8306c9fc-6af9-450f-9c70-47b912bc34a1\") " Nov 01 01:15:19.776258 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.776235 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8306c9fc-6af9-450f-9c70-47b912bc34a1-maas-api-service-ca" pod "8306c9fc-6af9-450f-9c70-47b912bc34a1" (UID: "8306c9fc-6af9-450f-9c70-47b912bc34a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:15:19.776353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.776335 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8306c9fc-6af9-450f-9c70-47b912bc34a1-maas-api-service-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:15:19.778002 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.777977 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8306c9fc-6af9-450f-9c70-47b912bc34a1-kube-api-access-92kpj" pod "8306c9fc-6af9-450f-9c70-47b912bc34a1" (UID: "8306c9fc-6af9-450f-9c70-47b912bc34a1"). InnerVolumeSpecName "kube-api-access-92kpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:15:19.877749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:19.877578 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92kpj\" (UniqueName: \"kubernetes.io/projected/8306c9fc-6af9-450f-9c70-47b912bc34a1-kube-api-access-92kpj\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:15:20.190922 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:20.190847 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" path="/var/lib/kubelet/pods/5afdb75f-def1-43d8-811e-3fbfa7afa195/volumes" Nov 01 01:15:20.385079 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:20.385054 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" Nov 01 01:15:20.385262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:20.385092 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk" event={"ID":"8306c9fc-6af9-450f-9c70-47b912bc34a1","Type":"ContainerDied","Data":"ea5f085c08b770bdd3ef739b508d895b3d1230d9de795e9a30f7af9d8e630299"} Nov 01 01:15:20.385262 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:20.385156 4910 scope.go:122] "RemoveContainer" containerID="39034b9589c622a6e9e6e196b718cbcb19f672b5ad82c71a430ee5080e19d403" Nov 01 01:15:20.553194 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:20.553158 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk"] Nov 01 01:15:20.602854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:20.602814 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7fdf2c-29891595-m8gfk"] Nov 01 01:15:22.191464 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:22.191432 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" path="/var/lib/kubelet/pods/8306c9fc-6af9-450f-9c70-47b912bc34a1/volumes" Nov 01 01:15:25.056791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:25.056757 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs"] Nov 01 01:15:25.107071 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:25.107039 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46ec952-m6bqs"] Nov 01 01:15:26.191115 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:26.191085 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339fac2d-201a-4c40-b722-366e74518ce4" path="/var/lib/kubelet/pods/339fac2d-201a-4c40-b722-366e74518ce4/volumes" Nov 01 01:15:33.216740 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:15:33.216692 4910 scope.go:122] "RemoveContainer" containerID="a723cb6b3e803a4d810a36bc8be73c3e993d3f031f7014d4d6aa5d2f48a4ebfe" Nov 01 01:15:43.102495 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:15:43.102457 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:16:13.105108 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:16:13.105073 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:16:15.058709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:15.058669 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr"] Nov 01 01:16:15.108291 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:15.108243 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7579f87a-6lzxr"] Nov 01 01:16:16.191077 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:16.191040 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a51e047-9bbd-450c-898b-86c9e79781bd" path="/var/lib/kubelet/pods/2a51e047-9bbd-450c-898b-86c9e79781bd/volumes" Nov 01 01:16:33.250045 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:33.250010 4910 scope.go:122] "RemoveContainer" containerID="e69c4b9c2f3520358232a3a03aba78b0294619541707313d861aba596eccdfb9" Nov 01 01:16:33.263091 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:33.263066 4910 scope.go:122] "RemoveContainer" containerID="dfbdfd7b185f675517dcb96611081401282527d82a4c519cff069484ddb107bf" Nov 01 01:16:33.272582 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:33.272559 4910 scope.go:122] "RemoveContainer" containerID="d27b57c2df759a2605a9d1cebd353de69d4fbb7892b4eb6939239c6f7f03fe80" Nov 01 01:16:33.281172 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:33.281154 4910 scope.go:122] "RemoveContainer" containerID="586f58a2482a6fa429084a89d583a19f93a7c8d12c20ed9e560e3665cdf049c5" Nov 01 01:16:33.289221 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:33.289194 4910 scope.go:122] "RemoveContainer" containerID="5abebda7e9ef87f83c60634f0564e56e2f07b3ec2c0d646e595b3555ecdfb082" Nov 01 01:16:43.102545 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:16:43.102465 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:16:49.594878 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.594835 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq"] Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595323 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595338 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595350 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595356 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595370 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595375 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595381 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595386 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595402 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:16:49.595424 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595408 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:16:49.595810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595503 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:16:49.595810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595516 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595522 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595529 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afdb75f-def1-43d8-811e-3fbfa7afa195" containerName="cleanup" Nov 01 01:16:49.595810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595533 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:16:49.595810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.595539 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:16:49.600578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.600562 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.656369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.656342 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq"] Nov 01 01:16:49.702675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.702644 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:16:49.702675 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.702665 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:16:49.751791 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.751754 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxlf\" (UniqueName: \"kubernetes.io/projected/2da41182-38b8-48d1-b83c-c81680dc95f5-kube-api-access-ggxlf\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.752008 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.751812 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.752008 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.751948 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.853065 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.852979 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.853229 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.853064 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.853229 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.853134 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxlf\" (UniqueName: \"kubernetes.io/projected/2da41182-38b8-48d1-b83c-c81680dc95f5-kube-api-access-ggxlf\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.853809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.853780 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.855711 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.855684 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:49.915000 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:49.914973 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxlf\" (UniqueName: \"kubernetes.io/projected/2da41182-38b8-48d1-b83c-c81680dc95f5-kube-api-access-ggxlf\") pod \"maas-api-revoke-sub-1531a9a1-wd8hq\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:50.214155 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:50.214075 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:50.594755 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:50.594724 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq"] Nov 01 01:16:50.595793 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:16:50.595768 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da41182_38b8_48d1_b83c_c81680dc95f5.slice/crio-3f76830688e934be5cb3334f046eec76fb65b29de850d26af160b43cca8fbf23 WatchSource:0}: Error finding container 3f76830688e934be5cb3334f046eec76fb65b29de850d26af160b43cca8fbf23: Status 404 returned error can't find the container with id 3f76830688e934be5cb3334f046eec76fb65b29de850d26af160b43cca8fbf23 Nov 01 01:16:50.954705 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:50.954661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" event={"ID":"2da41182-38b8-48d1-b83c-c81680dc95f5","Type":"ContainerStarted","Data":"694cd0755d0ca5d298af084e504a9d13b99d0c15b79450026eabbf5abbeecbc9"} Nov 01 01:16:50.954875 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:50.954760 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" event={"ID":"2da41182-38b8-48d1-b83c-c81680dc95f5","Type":"ContainerStarted","Data":"3f76830688e934be5cb3334f046eec76fb65b29de850d26af160b43cca8fbf23"} Nov 01 01:16:51.071592 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:51.071541 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" podStartSLOduration=2.071527307 podStartE2EDuration="2.071527307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:16:51.070379162 +0000 UTC m=+3201.049961497" watchObservedRunningTime="2026-11-01 01:16:51.071527307 +0000 UTC m=+3201.051109665" Nov 01 01:16:51.962928 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:51.962887 4910 generic.go:356] "Generic (PLEG): container finished" podID="2da41182-38b8-48d1-b83c-c81680dc95f5" containerID="694cd0755d0ca5d298af084e504a9d13b99d0c15b79450026eabbf5abbeecbc9" exitCode=0 Nov 01 01:16:51.963360 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:51.962965 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" event={"ID":"2da41182-38b8-48d1-b83c-c81680dc95f5","Type":"ContainerDied","Data":"694cd0755d0ca5d298af084e504a9d13b99d0c15b79450026eabbf5abbeecbc9"} Nov 01 01:16:53.112951 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.112931 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:53.188126 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.188096 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-xks-ca\") pod \"2da41182-38b8-48d1-b83c-c81680dc95f5\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " Nov 01 01:16:53.188289 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.188200 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da41182-38b8-48d1-b83c-c81680dc95f5-kube-api-access-ggxlf\" (UniqueName: \"kubernetes.io/projected/2da41182-38b8-48d1-b83c-c81680dc95f5-kube-api-access-ggxlf\") pod \"2da41182-38b8-48d1-b83c-c81680dc95f5\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " Nov 01 01:16:53.188289 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.188260 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-openshift-ca\") pod \"2da41182-38b8-48d1-b83c-c81680dc95f5\" (UID: \"2da41182-38b8-48d1-b83c-c81680dc95f5\") " Nov 01 01:16:53.188673 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.188651 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-openshift-ca" pod "2da41182-38b8-48d1-b83c-c81680dc95f5" (UID: "2da41182-38b8-48d1-b83c-c81680dc95f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:16:53.190304 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.190277 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-xks-ca" pod "2da41182-38b8-48d1-b83c-c81680dc95f5" (UID: "2da41182-38b8-48d1-b83c-c81680dc95f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:16:53.190452 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.190433 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da41182-38b8-48d1-b83c-c81680dc95f5-kube-api-access-ggxlf" pod "2da41182-38b8-48d1-b83c-c81680dc95f5" (UID: "2da41182-38b8-48d1-b83c-c81680dc95f5"). InnerVolumeSpecName "kube-api-access-ggxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:16:53.289137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.289106 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:53.289137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.289133 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da41182-38b8-48d1-b83c-c81680dc95f5-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:53.289137 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.289143 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxlf\" (UniqueName: \"kubernetes.io/projected/2da41182-38b8-48d1-b83c-c81680dc95f5-kube-api-access-ggxlf\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:16:53.976862 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.976831 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" Nov 01 01:16:53.977059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.976862 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq" event={"ID":"2da41182-38b8-48d1-b83c-c81680dc95f5","Type":"ContainerDied","Data":"3f76830688e934be5cb3334f046eec76fb65b29de850d26af160b43cca8fbf23"} Nov 01 01:16:53.977059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:16:53.976910 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f76830688e934be5cb3334f046eec76fb65b29de850d26af160b43cca8fbf23" Nov 01 01:17:02.816519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.816488 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w"] Nov 01 01:17:02.816960 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.816946 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2da41182-38b8-48d1-b83c-c81680dc95f5" containerName="revoke-keys" Nov 01 01:17:02.817011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.816960 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da41182-38b8-48d1-b83c-c81680dc95f5" containerName="revoke-keys" Nov 01 01:17:02.817011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.816983 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:17:02.817011 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.816989 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8306c9fc-6af9-450f-9c70-47b912bc34a1" containerName="cleanup" Nov 01 01:17:02.817126 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.817078 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da41182-38b8-48d1-b83c-c81680dc95f5" containerName="revoke-keys" Nov 01 01:17:02.820267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.820249 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:02.874482 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.874457 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:17:02.878527 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.878503 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w"] Nov 01 01:17:02.902908 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.902878 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59f72\" (UniqueName: \"kubernetes.io/projected/f5f079a9-3cf9-4ae2-be11-0754f442141b-kube-api-access-59f72\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:02.903064 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.902967 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:02.903064 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.903055 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:02.924353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:02.924330 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:17:03.003789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.003758 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:03.003958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.003807 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59f72\" (UniqueName: \"kubernetes.io/projected/f5f079a9-3cf9-4ae2-be11-0754f442141b-kube-api-access-59f72\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:03.003958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.003862 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:03.004421 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.004402 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:03.006343 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.006319 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:03.067399 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.067320 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59f72\" (UniqueName: \"kubernetes.io/projected/f5f079a9-3cf9-4ae2-be11-0754f442141b-kube-api-access-59f72\") pod \"maas-api-revoke-sub-5350c629-nkz9w\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:03.133508 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.133481 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:03.509066 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:03.509036 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w"] Nov 01 01:17:03.510581 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:17:03.510549 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f079a9_3cf9_4ae2_be11_0754f442141b.slice/crio-8dcfc36e49d6797adef2e46fa7356d2f97cd88d3de83b9a2cfc07a59be7d0ce7 WatchSource:0}: Error finding container 8dcfc36e49d6797adef2e46fa7356d2f97cd88d3de83b9a2cfc07a59be7d0ce7: Status 404 returned error can't find the container with id 8dcfc36e49d6797adef2e46fa7356d2f97cd88d3de83b9a2cfc07a59be7d0ce7 Nov 01 01:17:04.042611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:04.042519 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" event={"ID":"f5f079a9-3cf9-4ae2-be11-0754f442141b","Type":"ContainerStarted","Data":"4403db66eb58f5172e72cce9dba4d284d0197a338eff4d17be6d88c74213f331"} Nov 01 01:17:04.042611 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:04.042583 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" event={"ID":"f5f079a9-3cf9-4ae2-be11-0754f442141b","Type":"ContainerStarted","Data":"8dcfc36e49d6797adef2e46fa7356d2f97cd88d3de83b9a2cfc07a59be7d0ce7"} Nov 01 01:17:04.162770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:04.162719 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" podStartSLOduration=2.16270283 podStartE2EDuration="2.16270283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:17:04.162180714 +0000 UTC m=+3214.141763039" watchObservedRunningTime="2026-11-01 01:17:04.16270283 +0000 UTC m=+3214.142285161" Nov 01 01:17:05.054212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:05.054172 4910 generic.go:356] "Generic (PLEG): container finished" podID="f5f079a9-3cf9-4ae2-be11-0754f442141b" containerID="4403db66eb58f5172e72cce9dba4d284d0197a338eff4d17be6d88c74213f331" exitCode=0 Nov 01 01:17:05.054657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:05.054266 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" event={"ID":"f5f079a9-3cf9-4ae2-be11-0754f442141b","Type":"ContainerDied","Data":"4403db66eb58f5172e72cce9dba4d284d0197a338eff4d17be6d88c74213f331"} Nov 01 01:17:06.213491 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.213467 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:06.233542 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.233518 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-openshift-ca\") pod \"f5f079a9-3cf9-4ae2-be11-0754f442141b\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " Nov 01 01:17:06.233702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.233575 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f079a9-3cf9-4ae2-be11-0754f442141b-kube-api-access-59f72\" (UniqueName: \"kubernetes.io/projected/f5f079a9-3cf9-4ae2-be11-0754f442141b-kube-api-access-59f72\") pod \"f5f079a9-3cf9-4ae2-be11-0754f442141b\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " Nov 01 01:17:06.233702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.233601 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-xks-ca\") pod \"f5f079a9-3cf9-4ae2-be11-0754f442141b\" (UID: \"f5f079a9-3cf9-4ae2-be11-0754f442141b\") " Nov 01 01:17:06.233969 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.233941 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-openshift-ca" pod "f5f079a9-3cf9-4ae2-be11-0754f442141b" (UID: "f5f079a9-3cf9-4ae2-be11-0754f442141b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:17:06.235804 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.235782 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-xks-ca" pod "f5f079a9-3cf9-4ae2-be11-0754f442141b" (UID: "f5f079a9-3cf9-4ae2-be11-0754f442141b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:17:06.235914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.235900 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f079a9-3cf9-4ae2-be11-0754f442141b-kube-api-access-59f72" pod "f5f079a9-3cf9-4ae2-be11-0754f442141b" (UID: "f5f079a9-3cf9-4ae2-be11-0754f442141b"). InnerVolumeSpecName "kube-api-access-59f72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:17:06.334836 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.334759 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59f72\" (UniqueName: \"kubernetes.io/projected/f5f079a9-3cf9-4ae2-be11-0754f442141b-kube-api-access-59f72\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:06.334836 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.334788 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:06.334836 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:06.334801 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f079a9-3cf9-4ae2-be11-0754f442141b-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:07.070039 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:07.070001 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" event={"ID":"f5f079a9-3cf9-4ae2-be11-0754f442141b","Type":"ContainerDied","Data":"8dcfc36e49d6797adef2e46fa7356d2f97cd88d3de83b9a2cfc07a59be7d0ce7"} Nov 01 01:17:07.070222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:07.070051 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dcfc36e49d6797adef2e46fa7356d2f97cd88d3de83b9a2cfc07a59be7d0ce7" Nov 01 01:17:07.070222 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:07.070010 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w" Nov 01 01:17:13.102699 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:17:13.102664 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:17:26.059672 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:26.059617 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp"] Nov 01 01:17:26.107753 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:26.107718 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be015e3-92hvp"] Nov 01 01:17:26.190958 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:26.190922 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7324e8-e1af-4561-aa01-6fcee3cb4717" path="/var/lib/kubelet/pods/ef7324e8-e1af-4561-aa01-6fcee3cb4717/volumes" Nov 01 01:17:27.047794 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:27.047758 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t"] Nov 01 01:17:27.097195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:27.097160 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05f7811-fm22t"] Nov 01 01:17:28.194144 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:28.194107 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1992a42f-a61e-4234-b4c0-2d1a21494785" path="/var/lib/kubelet/pods/1992a42f-a61e-4234-b4c0-2d1a21494785/volumes" Nov 01 01:17:30.713832 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:30.713799 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw"] Nov 01 01:17:31.269338 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.269308 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5"] Nov 01 01:17:31.269876 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.269841 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f5f079a9-3cf9-4ae2-be11-0754f442141b" containerName="revoke-keys" Nov 01 01:17:31.269876 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.269863 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f079a9-3cf9-4ae2-be11-0754f442141b" containerName="revoke-keys" Nov 01 01:17:31.270070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.270021 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f079a9-3cf9-4ae2-be11-0754f442141b" containerName="revoke-keys" Nov 01 01:17:31.273253 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.273236 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.328741 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.328713 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:17:31.328741 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.328735 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:17:31.332772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.332752 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5"] Nov 01 01:17:31.399850 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.399818 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.399993 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.399873 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.400049 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.400031 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlhdq\" (UniqueName: \"kubernetes.io/projected/62c0babd-e8d9-47ea-a74e-4e7fe068802a-kube-api-access-tlhdq\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.500515 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.500482 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlhdq\" (UniqueName: \"kubernetes.io/projected/62c0babd-e8d9-47ea-a74e-4e7fe068802a-kube-api-access-tlhdq\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.500684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.500520 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.500684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.500567 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.501306 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.501281 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.503320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.503295 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.562829 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.562765 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlhdq\" (UniqueName: \"kubernetes.io/projected/62c0babd-e8d9-47ea-a74e-4e7fe068802a-kube-api-access-tlhdq\") pod \"maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.586385 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.586353 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:31.962512 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:17:31.962483 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c0babd_e8d9_47ea_a74e_4e7fe068802a.slice/crio-745d4381e5d737ffd0a5d3c9e1e3bdf7fbc910d9fd76b8cfec91c5d5499870a9 WatchSource:0}: Error finding container 745d4381e5d737ffd0a5d3c9e1e3bdf7fbc910d9fd76b8cfec91c5d5499870a9: Status 404 returned error can't find the container with id 745d4381e5d737ffd0a5d3c9e1e3bdf7fbc910d9fd76b8cfec91c5d5499870a9 Nov 01 01:17:31.962983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:31.962920 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5"] Nov 01 01:17:32.228963 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:32.228893 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" event={"ID":"62c0babd-e8d9-47ea-a74e-4e7fe068802a","Type":"ContainerStarted","Data":"745d4381e5d737ffd0a5d3c9e1e3bdf7fbc910d9fd76b8cfec91c5d5499870a9"} Nov 01 01:17:33.238617 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:33.238529 4910 generic.go:356] "Generic (PLEG): container finished" podID="62c0babd-e8d9-47ea-a74e-4e7fe068802a" containerID="b9590152e7bd16e307d0afb7e1ed06ce04abdc80c8e862544f358c0850feff34" exitCode=0 Nov 01 01:17:33.239031 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:33.238634 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" event={"ID":"62c0babd-e8d9-47ea-a74e-4e7fe068802a","Type":"ContainerDied","Data":"b9590152e7bd16e307d0afb7e1ed06ce04abdc80c8e862544f358c0850feff34"} Nov 01 01:17:33.328039 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:33.328009 4910 scope.go:122] "RemoveContainer" containerID="a906b8ca8d03cdfe1350abec331e996fca1098e16260ba65ce220a0e6319d6f4" Nov 01 01:17:33.337595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:33.337578 4910 scope.go:122] "RemoveContainer" containerID="93bd2e33e43f0087b568311a2080a547c6ff40a135e9157cb7110221e3df7257" Nov 01 01:17:34.382657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.382635 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:34.429967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.429934 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-xks-ca\") pod \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " Nov 01 01:17:34.430111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.430029 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0babd-e8d9-47ea-a74e-4e7fe068802a-kube-api-access-tlhdq\" (UniqueName: \"kubernetes.io/projected/62c0babd-e8d9-47ea-a74e-4e7fe068802a-kube-api-access-tlhdq\") pod \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " Nov 01 01:17:34.430111 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.430075 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-openshift-ca\") pod \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\" (UID: \"62c0babd-e8d9-47ea-a74e-4e7fe068802a\") " Nov 01 01:17:34.430514 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.430489 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-openshift-ca" pod "62c0babd-e8d9-47ea-a74e-4e7fe068802a" (UID: "62c0babd-e8d9-47ea-a74e-4e7fe068802a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:17:34.432058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.432036 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-xks-ca" pod "62c0babd-e8d9-47ea-a74e-4e7fe068802a" (UID: "62c0babd-e8d9-47ea-a74e-4e7fe068802a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:17:34.432158 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.432099 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0babd-e8d9-47ea-a74e-4e7fe068802a-kube-api-access-tlhdq" pod "62c0babd-e8d9-47ea-a74e-4e7fe068802a" (UID: "62c0babd-e8d9-47ea-a74e-4e7fe068802a"). InnerVolumeSpecName "kube-api-access-tlhdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:17:34.531036 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.531003 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlhdq\" (UniqueName: \"kubernetes.io/projected/62c0babd-e8d9-47ea-a74e-4e7fe068802a-kube-api-access-tlhdq\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:34.531180 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.531043 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:34.531180 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:34.531061 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c0babd-e8d9-47ea-a74e-4e7fe068802a-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:35.252206 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:35.252171 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" Nov 01 01:17:35.252206 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:35.252199 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5" event={"ID":"62c0babd-e8d9-47ea-a74e-4e7fe068802a","Type":"ContainerDied","Data":"745d4381e5d737ffd0a5d3c9e1e3bdf7fbc910d9fd76b8cfec91c5d5499870a9"} Nov 01 01:17:35.252437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:35.252248 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="745d4381e5d737ffd0a5d3c9e1e3bdf7fbc910d9fd76b8cfec91c5d5499870a9" Nov 01 01:17:43.107117 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:17:43.107078 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:17:45.734322 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:45.734282 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" podUID="52171400-83a4-4fe2-b2dc-34504dbc3259" containerName="main" containerID="cri-o://c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c" gracePeriod=45 Nov 01 01:17:45.996554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:45.996529 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:17:46.038896 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.038867 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-tmp-dir\") pod \"52171400-83a4-4fe2-b2dc-34504dbc3259\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " Nov 01 01:17:46.039059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.038913 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-dshm\") pod \"52171400-83a4-4fe2-b2dc-34504dbc3259\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " Nov 01 01:17:46.039059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.038950 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-model-cache\") pod \"52171400-83a4-4fe2-b2dc-34504dbc3259\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " Nov 01 01:17:46.039059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.038998 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52171400-83a4-4fe2-b2dc-34504dbc3259-tls-certs\" (UniqueName: \"kubernetes.io/secret/52171400-83a4-4fe2-b2dc-34504dbc3259-tls-certs\") pod \"52171400-83a4-4fe2-b2dc-34504dbc3259\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " Nov 01 01:17:46.039059 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039031 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52171400-83a4-4fe2-b2dc-34504dbc3259-kube-api-access-frh7d\" (UniqueName: \"kubernetes.io/projected/52171400-83a4-4fe2-b2dc-34504dbc3259-kube-api-access-frh7d\") pod \"52171400-83a4-4fe2-b2dc-34504dbc3259\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " Nov 01 01:17:46.039292 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039129 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-tmp-dir" pod "52171400-83a4-4fe2-b2dc-34504dbc3259" (UID: "52171400-83a4-4fe2-b2dc-34504dbc3259"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:17:46.039292 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039159 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-home\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-home\") pod \"52171400-83a4-4fe2-b2dc-34504dbc3259\" (UID: \"52171400-83a4-4fe2-b2dc-34504dbc3259\") " Nov 01 01:17:46.039292 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039242 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-model-cache" pod "52171400-83a4-4fe2-b2dc-34504dbc3259" (UID: "52171400-83a4-4fe2-b2dc-34504dbc3259"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:17:46.039461 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039396 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-home" pod "52171400-83a4-4fe2-b2dc-34504dbc3259" (UID: "52171400-83a4-4fe2-b2dc-34504dbc3259"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:17:46.039566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039549 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:46.039655 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039569 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:46.039655 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.039579 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:46.041261 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.041232 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-dshm" pod "52171400-83a4-4fe2-b2dc-34504dbc3259" (UID: "52171400-83a4-4fe2-b2dc-34504dbc3259"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:17:46.041833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.041807 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52171400-83a4-4fe2-b2dc-34504dbc3259-tls-certs" pod "52171400-83a4-4fe2-b2dc-34504dbc3259" (UID: "52171400-83a4-4fe2-b2dc-34504dbc3259"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:17:46.041907 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.041838 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52171400-83a4-4fe2-b2dc-34504dbc3259-kube-api-access-frh7d" pod "52171400-83a4-4fe2-b2dc-34504dbc3259" (UID: "52171400-83a4-4fe2-b2dc-34504dbc3259"). InnerVolumeSpecName "kube-api-access-frh7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:17:46.141063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.141033 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52171400-83a4-4fe2-b2dc-34504dbc3259-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:46.141063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.141059 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frh7d\" (UniqueName: \"kubernetes.io/projected/52171400-83a4-4fe2-b2dc-34504dbc3259-kube-api-access-frh7d\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:46.141063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.141068 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52171400-83a4-4fe2-b2dc-34504dbc3259-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:17:46.326067 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.325992 4910 generic.go:356] "Generic (PLEG): container finished" podID="52171400-83a4-4fe2-b2dc-34504dbc3259" containerID="c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c" exitCode=0 Nov 01 01:17:46.326067 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.326038 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" Nov 01 01:17:46.326067 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.326060 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" event={"ID":"52171400-83a4-4fe2-b2dc-34504dbc3259","Type":"ContainerDied","Data":"c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c"} Nov 01 01:17:46.326340 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.326092 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" event={"ID":"52171400-83a4-4fe2-b2dc-34504dbc3259","Type":"ContainerDied","Data":"dd65306df8d3f4c48c8cc6dceb8281b5387a56de578be8de164cfcc8a766da55"} Nov 01 01:17:46.326340 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.326108 4910 scope.go:122] "RemoveContainer" containerID="c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c" Nov 01 01:17:46.335712 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.335691 4910 scope.go:122] "RemoveContainer" containerID="c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c" Nov 01 01:17:46.336008 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:17:46.335988 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c\": container with ID starting with c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c not found: ID does not exist" containerID="c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c" Nov 01 01:17:46.336079 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.336020 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c"} err="failed to get container status \"c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c\": rpc error: code = NotFound desc = could not find container \"c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c\": container with ID starting with c22bb7f2f20fafeb702f7706a07dd89c322787e591e4fad2d73cb14c6e185e2c not found: ID does not exist" Nov 01 01:17:46.493093 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.493066 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw"] Nov 01 01:17:46.499416 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.499393 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw"] Nov 01 01:17:46.946058 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:46.946020 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-48d289/test-model-48d289-kserve-647c8c6d45-rzlrw" podUID="52171400-83a4-4fe2-b2dc-34504dbc3259" containerName="main" probeResult="failure" output="Get \"https://10.132.0.229:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 01:17:48.190711 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:48.190679 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52171400-83a4-4fe2-b2dc-34504dbc3259" path="/var/lib/kubelet/pods/52171400-83a4-4fe2-b2dc-34504dbc3259/volumes" Nov 01 01:17:50.621320 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:50.621283 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5"] Nov 01 01:17:50.671130 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:50.671099 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7fdf2c-vkzc5"] Nov 01 01:17:52.191297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:52.191255 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c0babd-e8d9-47ea-a74e-4e7fe068802a" path="/var/lib/kubelet/pods/62c0babd-e8d9-47ea-a74e-4e7fe068802a/volumes" Nov 01 01:17:57.727280 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:57.727247 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml"] Nov 01 01:17:57.727846 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:57.727446 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" containerID="cri-o://9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112" gracePeriod=30 Nov 01 01:17:58.050036 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:58.050005 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v"] Nov 01 01:17:58.054526 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:58.054496 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt"] Nov 01 01:17:58.057369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:58.057349 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd0adb0-mbd2v"] Nov 01 01:17:58.059578 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:58.059558 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049c6ebd-7zkkt"] Nov 01 01:17:58.191649 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:58.191596 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0271ed6d-f127-4a5b-a01c-b909c4ada584" path="/var/lib/kubelet/pods/0271ed6d-f127-4a5b-a01c-b909c4ada584/volumes" Nov 01 01:17:58.192068 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:58.192054 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ae261e-6b6e-4c5b-b72a-61732e457371" path="/var/lib/kubelet/pods/e3ae261e-6b6e-4c5b-b72a-61732e457371/volumes" Nov 01 01:17:59.271709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:59.271671 4910 patch_prober.go:29] interesting pod/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 01:17:59.272175 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:17:59.271729 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 01:18:02.986533 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:02.986506 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:18:03.115393 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115358 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809745fa-9b1c-4c29-8b67-8162926139db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809745fa-9b1c-4c29-8b67-8162926139db-istiod-ca-cert\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115401 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-istio-token\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-istio-token\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115468 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-certs\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115529 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-socket\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115567 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115565 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-data\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115581 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-envoy\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115681 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-credential-socket\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115710 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/809745fa-9b1c-4c29-8b67-8162926139db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809745fa-9b1c-4c29-8b67-8162926139db-istio-podinfo\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.115854 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115755 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-kube-api-access-t8jcw\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-kube-api-access-t8jcw\") pod \"809745fa-9b1c-4c29-8b67-8162926139db\" (UID: \"809745fa-9b1c-4c29-8b67-8162926139db\") " Nov 01 01:18:03.116084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115901 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-data" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.116084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115913 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-socket" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.116084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115915 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809745fa-9b1c-4c29-8b67-8162926139db-istiod-ca-cert" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:03.116084 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.115927 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-certs" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.116469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.116115 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-credential-socket" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.116469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.116184 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/809745fa-9b1c-4c29-8b67-8162926139db-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.116469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.116204 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.116469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.116216 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.116469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.116228 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.118053 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.118029 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-istio-token" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:03.118164 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.118049 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-envoy" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:03.118164 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.118051 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-kube-api-access-t8jcw" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "kube-api-access-t8jcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:03.118164 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.118110 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/809745fa-9b1c-4c29-8b67-8162926139db-istio-podinfo" pod "809745fa-9b1c-4c29-8b67-8162926139db" (UID: "809745fa-9b1c-4c29-8b67-8162926139db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:18:03.217046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.217010 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8jcw\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-kube-api-access-t8jcw\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.217213 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.217058 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/809745fa-9b1c-4c29-8b67-8162926139db-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.217213 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.217074 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.217213 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.217088 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/809745fa-9b1c-4c29-8b67-8162926139db-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.217213 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.217103 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/809745fa-9b1c-4c29-8b67-8162926139db-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:03.436530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.436443 4910 generic.go:356] "Generic (PLEG): container finished" podID="809745fa-9b1c-4c29-8b67-8162926139db" containerID="9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112" exitCode=0 Nov 01 01:18:03.436530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.436496 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" Nov 01 01:18:03.436831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.436537 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" event={"ID":"809745fa-9b1c-4c29-8b67-8162926139db","Type":"ContainerDied","Data":"9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112"} Nov 01 01:18:03.436831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.436603 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml" event={"ID":"809745fa-9b1c-4c29-8b67-8162926139db","Type":"ContainerDied","Data":"d45dc63db0554215a8b22f1bb0a7d5cfdf8c8d8aed5b2eb6c28b36e1d8927365"} Nov 01 01:18:03.436831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.436665 4910 scope.go:122] "RemoveContainer" containerID="9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112" Nov 01 01:18:03.449737 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.449715 4910 scope.go:122] "RemoveContainer" containerID="9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112" Nov 01 01:18:03.450066 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:18:03.450044 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112\": container with ID starting with 9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112 not found: ID does not exist" containerID="9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112" Nov 01 01:18:03.450136 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.450080 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112"} err="failed to get container status \"9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112\": rpc error: code = NotFound desc = could not find container \"9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112\": container with ID starting with 9d850e6b157dced33cac5abbe0aa22b757ade195845501d112560419bfbe1112 not found: ID does not exist" Nov 01 01:18:03.602551 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.602512 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml"] Nov 01 01:18:03.652440 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:03.652409 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7fdf2c-openshift-default-bbb94cfd7-tfkml"] Nov 01 01:18:04.192070 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:04.192030 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809745fa-9b1c-4c29-8b67-8162926139db" path="/var/lib/kubelet/pods/809745fa-9b1c-4c29-8b67-8162926139db/volumes" Nov 01 01:18:13.102444 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:18:13.102408 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:18:23.177965 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:23.177929 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft"] Nov 01 01:18:23.178450 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:23.178155 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" containerID="cri-o://5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd" gracePeriod=30 Nov 01 01:18:28.430983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.430940 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:18:28.461518 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461454 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-certs\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461518 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461489 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-data\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461518 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461518 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d40aac9-e330-4270-8d4e-5770a83b105e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40aac9-e330-4270-8d4e-5770a83b105e-istiod-ca-cert\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461562 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-kube-api-access-h5l62\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-kube-api-access-h5l62\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461613 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-podinfo\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461686 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-socket\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461728 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-credential-socket\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461774 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-token\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-token\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.461858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461780 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-data" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:28.461858 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461820 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-certs" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:28.462245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.461860 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-envoy\") pod \"5d40aac9-e330-4270-8d4e-5770a83b105e\" (UID: \"5d40aac9-e330-4270-8d4e-5770a83b105e\") " Nov 01 01:18:28.462245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462022 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-socket" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:28.462245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462049 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-credential-socket" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:28.462245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462063 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d40aac9-e330-4270-8d4e-5770a83b105e-istiod-ca-cert" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:28.462489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462305 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.462489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462328 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.462489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462343 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40aac9-e330-4270-8d4e-5770a83b105e-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.462489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462358 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.462489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.462373 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.464132 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.464103 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-kube-api-access-h5l62" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "kube-api-access-h5l62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:28.464266 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.464243 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-token" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:28.464311 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.464277 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-envoy" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:28.464311 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.464280 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-podinfo" pod "5d40aac9-e330-4270-8d4e-5770a83b105e" (UID: "5d40aac9-e330-4270-8d4e-5770a83b105e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:18:28.563005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.562972 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5l62\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-kube-api-access-h5l62\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.563005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.563000 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.563176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.563016 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.563176 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.563028 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40aac9-e330-4270-8d4e-5770a83b105e-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:28.597054 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.597031 4910 generic.go:356] "Generic (PLEG): container finished" podID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerID="5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd" exitCode=0 Nov 01 01:18:28.597174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.597076 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" Nov 01 01:18:28.597174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.597120 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" event={"ID":"5d40aac9-e330-4270-8d4e-5770a83b105e","Type":"ContainerDied","Data":"5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd"} Nov 01 01:18:28.597251 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.597177 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft" event={"ID":"5d40aac9-e330-4270-8d4e-5770a83b105e","Type":"ContainerDied","Data":"94c7282e7a5bd3b3aafdaf946492368847640ef75bdf88201c0282bde6865d3b"} Nov 01 01:18:28.597251 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.597198 4910 scope.go:122] "RemoveContainer" containerID="5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd" Nov 01 01:18:28.609131 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.609114 4910 scope.go:122] "RemoveContainer" containerID="5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd" Nov 01 01:18:28.609491 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:18:28.609472 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd\": container with ID starting with 5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd not found: ID does not exist" containerID="5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd" Nov 01 01:18:28.609563 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.609501 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd"} err="failed to get container status \"5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd\": rpc error: code = NotFound desc = could not find container \"5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd\": container with ID starting with 5e997096b44176a259ee38a65ad4dde1b1182d9cf8bdbb9556e2fd8bdee67ccd not found: ID does not exist" Nov 01 01:18:28.773482 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.773449 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft"] Nov 01 01:18:28.777561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:28.777539 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pqmft"] Nov 01 01:18:29.623015 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:29.622983 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt"] Nov 01 01:18:29.639005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:29.638975 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5"] Nov 01 01:18:30.192351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:30.192319 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" path="/var/lib/kubelet/pods/5d40aac9-e330-4270-8d4e-5770a83b105e/volumes" Nov 01 01:18:30.960204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:30.960180 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:18:30.970049 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:30.970028 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:18:30.973593 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:30.973575 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:18:30.982945 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:30.982929 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:18:33.376278 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:33.376245 4910 scope.go:122] "RemoveContainer" containerID="1611df6415c41312a82e49623f3a36aac7d072d1c668523c7a0e800d3d4f1f05" Nov 01 01:18:33.385501 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:33.385473 4910 scope.go:122] "RemoveContainer" containerID="be5927e3215f5e65a6b7ed9d97d56543978f199299b05510cb479b65de75677a" Nov 01 01:18:33.394595 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:33.394578 4910 scope.go:122] "RemoveContainer" containerID="a7fe3144439c76b38c1ec84476e8241dffed92e611519af35e23f71bb36e1131" Nov 01 01:18:33.402609 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:33.402593 4910 scope.go:122] "RemoveContainer" containerID="ae3a545d101af62914bc5c6ff5213947241d4b493e81ba97b4d4e81f97fc1b60" Nov 01 01:18:36.442067 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442034 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m"] Nov 01 01:18:36.442495 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442444 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" Nov 01 01:18:36.442495 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442454 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" Nov 01 01:18:36.442495 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442467 4910 cpu_manager.go:414] "containerMap: removing container" podUID="62c0babd-e8d9-47ea-a74e-4e7fe068802a" containerName="revoke-keys" Nov 01 01:18:36.442495 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442473 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c0babd-e8d9-47ea-a74e-4e7fe068802a" containerName="revoke-keys" Nov 01 01:18:36.442495 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442486 4910 cpu_manager.go:414] "containerMap: removing container" podUID="52171400-83a4-4fe2-b2dc-34504dbc3259" containerName="main" Nov 01 01:18:36.442495 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442491 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52171400-83a4-4fe2-b2dc-34504dbc3259" containerName="main" Nov 01 01:18:36.442758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442507 4910 cpu_manager.go:414] "containerMap: removing container" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" Nov 01 01:18:36.442758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442512 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" Nov 01 01:18:36.442758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442583 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d40aac9-e330-4270-8d4e-5770a83b105e" containerName="istio-proxy" Nov 01 01:18:36.442758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442593 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c0babd-e8d9-47ea-a74e-4e7fe068802a" containerName="revoke-keys" Nov 01 01:18:36.442758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442602 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="52171400-83a4-4fe2-b2dc-34504dbc3259" containerName="main" Nov 01 01:18:36.442758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.442610 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="809745fa-9b1c-4c29-8b67-8162926139db" containerName="istio-proxy" Nov 01 01:18:36.447577 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.447561 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.506658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.506600 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m"] Nov 01 01:18:36.534468 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.534436 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.534610 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.534486 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.534729 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.534701 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89l52\" (UniqueName: \"kubernetes.io/projected/71c8a8e2-fc7c-4309-b54a-03f297b6326d-kube-api-access-89l52\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.553386 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.553356 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:18:36.553509 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.553392 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kdmzl\"" Nov 01 01:18:36.636096 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.636069 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.636266 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.636123 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.636266 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.636216 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l52\" (UniqueName: \"kubernetes.io/projected/71c8a8e2-fc7c-4309-b54a-03f297b6326d-kube-api-access-89l52\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.636738 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.636715 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.638618 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.638599 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.660735 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.660709 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d"] Nov 01 01:18:36.700583 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.700533 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89l52\" (UniqueName: \"kubernetes.io/projected/71c8a8e2-fc7c-4309-b54a-03f297b6326d-kube-api-access-89l52\") pod \"maas-api-revoke-sub-35ca337c-65c8m\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.760638 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.760596 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:36.771003 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:36.770978 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44"] Nov 01 01:18:37.143251 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:37.143221 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m"] Nov 01 01:18:37.144717 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:18:37.144692 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c8a8e2_fc7c_4309_b54a_03f297b6326d.slice/crio-2e96c82a9c7abc9bf5c0146ee60da75dc05a28872add534939408b8dbd19ed63 WatchSource:0}: Error finding container 2e96c82a9c7abc9bf5c0146ee60da75dc05a28872add534939408b8dbd19ed63: Status 404 returned error can't find the container with id 2e96c82a9c7abc9bf5c0146ee60da75dc05a28872add534939408b8dbd19ed63 Nov 01 01:18:37.657251 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:37.657216 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" event={"ID":"71c8a8e2-fc7c-4309-b54a-03f297b6326d","Type":"ContainerStarted","Data":"46e5291ec211f629982512b65dfc4d965181417c9de235d5f2b9be2181a994dd"} Nov 01 01:18:37.657702 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:37.657295 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" event={"ID":"71c8a8e2-fc7c-4309-b54a-03f297b6326d","Type":"ContainerStarted","Data":"2e96c82a9c7abc9bf5c0146ee60da75dc05a28872add534939408b8dbd19ed63"} Nov 01 01:18:37.775364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:37.775315 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" podStartSLOduration=1.7753027380000002 podStartE2EDuration="1.775302738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:18:37.772967629 +0000 UTC m=+3307.752549964" watchObservedRunningTime="2026-11-01 01:18:37.775302738 +0000 UTC m=+3307.754885073" Nov 01 01:18:38.299276 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:38.299242 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75"] Nov 01 01:18:38.299469 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:38.299435 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" containerID="cri-o://876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71" gracePeriod=30 Nov 01 01:18:38.664422 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:38.664382 4910 generic.go:356] "Generic (PLEG): container finished" podID="71c8a8e2-fc7c-4309-b54a-03f297b6326d" containerID="46e5291ec211f629982512b65dfc4d965181417c9de235d5f2b9be2181a994dd" exitCode=0 Nov 01 01:18:38.664851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:38.664478 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" event={"ID":"71c8a8e2-fc7c-4309-b54a-03f297b6326d","Type":"ContainerDied","Data":"46e5291ec211f629982512b65dfc4d965181417c9de235d5f2b9be2181a994dd"} Nov 01 01:18:39.824094 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.824070 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:39.967603 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.967525 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c8a8e2-fc7c-4309-b54a-03f297b6326d-kube-api-access-89l52\" (UniqueName: \"kubernetes.io/projected/71c8a8e2-fc7c-4309-b54a-03f297b6326d-kube-api-access-89l52\") pod \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " Nov 01 01:18:39.967817 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.967661 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-openshift-ca\") pod \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " Nov 01 01:18:39.967817 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.967695 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-xks-ca\") pod \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\" (UID: \"71c8a8e2-fc7c-4309-b54a-03f297b6326d\") " Nov 01 01:18:39.967955 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.967932 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-openshift-ca" pod "71c8a8e2-fc7c-4309-b54a-03f297b6326d" (UID: "71c8a8e2-fc7c-4309-b54a-03f297b6326d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:39.968062 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.968043 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:39.969894 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.969867 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-xks-ca" pod "71c8a8e2-fc7c-4309-b54a-03f297b6326d" (UID: "71c8a8e2-fc7c-4309-b54a-03f297b6326d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:18:39.970063 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:39.970038 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c8a8e2-fc7c-4309-b54a-03f297b6326d-kube-api-access-89l52" pod "71c8a8e2-fc7c-4309-b54a-03f297b6326d" (UID: "71c8a8e2-fc7c-4309-b54a-03f297b6326d"). InnerVolumeSpecName "kube-api-access-89l52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:40.068695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:40.068658 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c8a8e2-fc7c-4309-b54a-03f297b6326d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:40.068695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:40.068695 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89l52\" (UniqueName: \"kubernetes.io/projected/71c8a8e2-fc7c-4309-b54a-03f297b6326d-kube-api-access-89l52\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:40.679227 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:40.679198 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" Nov 01 01:18:40.679389 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:40.679241 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m" event={"ID":"71c8a8e2-fc7c-4309-b54a-03f297b6326d","Type":"ContainerDied","Data":"2e96c82a9c7abc9bf5c0146ee60da75dc05a28872add534939408b8dbd19ed63"} Nov 01 01:18:40.679389 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:40.679294 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e96c82a9c7abc9bf5c0146ee60da75dc05a28872add534939408b8dbd19ed63" Nov 01 01:18:41.661900 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:41.661860 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" podUID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" containerName="payload-processing" containerID="cri-o://91df9370a96e053d7b0a5ee094790b292c5bc505bec97eae3a4f448804ebac04" gracePeriod=25 Nov 01 01:18:41.771888 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:41.771846 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" podUID="1b98851f-c830-4d22-a98b-8e1de94bde25" containerName="payload-pre-processing" containerID="cri-o://6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b" gracePeriod=25 Nov 01 01:18:43.101872 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:18:43.101837 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:18:43.560069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.560043 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 01:18:43.698135 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.698051 4910 generic.go:356] "Generic (PLEG): container finished" podID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerID="876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71" exitCode=0 Nov 01 01:18:43.698135 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.698100 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" Nov 01 01:18:43.698367 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.698136 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" event={"ID":"dee64569-1fa7-4c78-b25f-32cefd90e832","Type":"ContainerDied","Data":"876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71"} Nov 01 01:18:43.698367 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.698218 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75" event={"ID":"dee64569-1fa7-4c78-b25f-32cefd90e832","Type":"ContainerDied","Data":"ab7d8f518e516a8535e6334659a64d1588601946b3fd61c4b8688ca81b20fb2f"} Nov 01 01:18:43.698367 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.698235 4910 scope.go:122] "RemoveContainer" containerID="876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71" Nov 01 01:18:43.704502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704481 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-podinfo\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704619 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704519 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee64569-1fa7-4c78-b25f-32cefd90e832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee64569-1fa7-4c78-b25f-32cefd90e832-istiod-ca-cert\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704619 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704544 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-envoy\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704619 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704605 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-socket\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704642 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-data\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704676 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-kube-api-access-9p5fm\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-kube-api-access-9p5fm\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704805 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704772 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-credential-socket\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704975 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704812 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-certs\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704975 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704842 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-token\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-token\") pod \"dee64569-1fa7-4c78-b25f-32cefd90e832\" (UID: \"dee64569-1fa7-4c78-b25f-32cefd90e832\") " Nov 01 01:18:43.704975 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704936 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-socket" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:43.704975 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.704956 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-data" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:43.705184 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.705003 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee64569-1fa7-4c78-b25f-32cefd90e832-istiod-ca-cert" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:18:43.705245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.705158 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-credential-socket" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:43.705245 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.705220 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee64569-1fa7-4c78-b25f-32cefd90e832-istiod-ca-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.705348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.705244 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.705348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.705258 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-data\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.705449 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.705377 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-certs" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:43.706853 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.706824 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-podinfo" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 01:18:43.707122 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.707102 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-token" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:43.707318 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.707299 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-envoy" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:43.707363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.707332 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-kube-api-access-9p5fm" pod "dee64569-1fa7-4c78-b25f-32cefd90e832" (UID: "dee64569-1fa7-4c78-b25f-32cefd90e832"). InnerVolumeSpecName "kube-api-access-9p5fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:43.708781 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.708755 4910 scope.go:122] "RemoveContainer" containerID="876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71" Nov 01 01:18:43.709065 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:18:43.709044 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71\": container with ID starting with 876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71 not found: ID does not exist" containerID="876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71" Nov 01 01:18:43.709133 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.709076 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71"} err="failed to get container status \"876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71\": rpc error: code = NotFound desc = could not find container \"876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71\": container with ID starting with 876dec2b1a3651291146111a8f48287615c171c83fa287a70bd1a21dbf074c71 not found: ID does not exist" Nov 01 01:18:43.806112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.806077 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p5fm\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-kube-api-access-9p5fm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.806112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.806102 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-credential-socket\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.806112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.806111 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-workload-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.806112 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.806119 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-token\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.806395 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.806128 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-podinfo\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:43.806395 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:43.806139 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee64569-1fa7-4c78-b25f-32cefd90e832-istio-envoy\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:44.127395 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.127364 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75"] Nov 01 01:18:44.177579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.177545 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-d6f75"] Nov 01 01:18:44.192707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.192676 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" path="/var/lib/kubelet/pods/dee64569-1fa7-4c78-b25f-32cefd90e832/volumes" Nov 01 01:18:44.646122 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.646084 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" containerID="cri-o://6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365" gracePeriod=45 Nov 01 01:18:44.659026 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.658984 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" containerID="cri-o://eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a" gracePeriod=45 Nov 01 01:18:44.807326 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.807272 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Nov 01 01:18:44.924298 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.924273 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 01:18:44.956046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:44.956023 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 01:18:45.016412 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016381 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-dshm\") pod \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " Nov 01 01:18:45.016566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016429 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-dshm\") pod \"46be65a6-4f18-4e60-855c-6b08571e2981\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " Nov 01 01:18:45.016566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016459 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-home\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-home\") pod \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " Nov 01 01:18:45.016566 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016524 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tmp-dir\") pod \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " Nov 01 01:18:45.016789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016585 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-home\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-home\") pod \"46be65a6-4f18-4e60-855c-6b08571e2981\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " Nov 01 01:18:45.016789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016614 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54f758e-a86e-44e8-aae6-eef45c5e6dda-kube-api-access-bdwss\" (UniqueName: \"kubernetes.io/projected/a54f758e-a86e-44e8-aae6-eef45c5e6dda-kube-api-access-bdwss\") pod \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " Nov 01 01:18:45.016789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016680 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-model-cache\") pod \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " Nov 01 01:18:45.016789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016724 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-model-cache\") pod \"46be65a6-4f18-4e60-855c-6b08571e2981\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " Nov 01 01:18:45.016789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016752 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tls-certs\" (UniqueName: \"kubernetes.io/secret/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tls-certs\") pod \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\" (UID: \"a54f758e-a86e-44e8-aae6-eef45c5e6dda\") " Nov 01 01:18:45.016789 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016781 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46be65a6-4f18-4e60-855c-6b08571e2981-kube-api-access-s2kdp\" (UniqueName: \"kubernetes.io/projected/46be65a6-4f18-4e60-855c-6b08571e2981-kube-api-access-s2kdp\") pod \"46be65a6-4f18-4e60-855c-6b08571e2981\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " Nov 01 01:18:45.017243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016805 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-home" pod "a54f758e-a86e-44e8-aae6-eef45c5e6dda" (UID: "a54f758e-a86e-44e8-aae6-eef45c5e6dda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.017243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016850 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-home" pod "46be65a6-4f18-4e60-855c-6b08571e2981" (UID: "46be65a6-4f18-4e60-855c-6b08571e2981"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.017243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016871 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46be65a6-4f18-4e60-855c-6b08571e2981-tls-certs\" (UniqueName: \"kubernetes.io/secret/46be65a6-4f18-4e60-855c-6b08571e2981-tls-certs\") pod \"46be65a6-4f18-4e60-855c-6b08571e2981\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " Nov 01 01:18:45.017243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016876 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tmp-dir" pod "a54f758e-a86e-44e8-aae6-eef45c5e6dda" (UID: "a54f758e-a86e-44e8-aae6-eef45c5e6dda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.017243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016978 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-tmp-dir\") pod \"46be65a6-4f18-4e60-855c-6b08571e2981\" (UID: \"46be65a6-4f18-4e60-855c-6b08571e2981\") " Nov 01 01:18:45.017243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016985 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-model-cache" pod "46be65a6-4f18-4e60-855c-6b08571e2981" (UID: "46be65a6-4f18-4e60-855c-6b08571e2981"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.017243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.016992 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-model-cache" pod "a54f758e-a86e-44e8-aae6-eef45c5e6dda" (UID: "a54f758e-a86e-44e8-aae6-eef45c5e6dda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.018243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.017215 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-tmp-dir" pod "46be65a6-4f18-4e60-855c-6b08571e2981" (UID: "46be65a6-4f18-4e60-855c-6b08571e2981"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.018243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.017525 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.018243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.017545 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.018243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.017560 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.018243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.017574 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.018243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.017588 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.018243 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.017603 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.019210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.019184 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-dshm" pod "46be65a6-4f18-4e60-855c-6b08571e2981" (UID: "46be65a6-4f18-4e60-855c-6b08571e2981"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.019496 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.019468 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tls-certs" pod "a54f758e-a86e-44e8-aae6-eef45c5e6dda" (UID: "a54f758e-a86e-44e8-aae6-eef45c5e6dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:18:45.019691 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.019668 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46be65a6-4f18-4e60-855c-6b08571e2981-tls-certs" pod "46be65a6-4f18-4e60-855c-6b08571e2981" (UID: "46be65a6-4f18-4e60-855c-6b08571e2981"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:18:45.019773 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.019753 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54f758e-a86e-44e8-aae6-eef45c5e6dda-kube-api-access-bdwss" pod "a54f758e-a86e-44e8-aae6-eef45c5e6dda" (UID: "a54f758e-a86e-44e8-aae6-eef45c5e6dda"). InnerVolumeSpecName "kube-api-access-bdwss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:45.019984 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.019954 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46be65a6-4f18-4e60-855c-6b08571e2981-kube-api-access-s2kdp" pod "46be65a6-4f18-4e60-855c-6b08571e2981" (UID: "46be65a6-4f18-4e60-855c-6b08571e2981"). InnerVolumeSpecName "kube-api-access-s2kdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:18:45.020329 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.020312 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-dshm" pod "a54f758e-a86e-44e8-aae6-eef45c5e6dda" (UID: "a54f758e-a86e-44e8-aae6-eef45c5e6dda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:18:45.120658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.118062 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46be65a6-4f18-4e60-855c-6b08571e2981-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.120658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.118090 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54f758e-a86e-44e8-aae6-eef45c5e6dda-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.120658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.118099 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46be65a6-4f18-4e60-855c-6b08571e2981-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.120658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.118111 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwss\" (UniqueName: \"kubernetes.io/projected/a54f758e-a86e-44e8-aae6-eef45c5e6dda-kube-api-access-bdwss\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.120658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.118139 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54f758e-a86e-44e8-aae6-eef45c5e6dda-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.120658 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.118152 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kdp\" (UniqueName: \"kubernetes.io/projected/46be65a6-4f18-4e60-855c-6b08571e2981-kube-api-access-s2kdp\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:18:45.711282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.711243 4910 generic.go:356] "Generic (PLEG): container finished" podID="46be65a6-4f18-4e60-855c-6b08571e2981" containerID="eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a" exitCode=0 Nov 01 01:18:45.711770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.711286 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" Nov 01 01:18:45.711770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.711325 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" event={"ID":"46be65a6-4f18-4e60-855c-6b08571e2981","Type":"ContainerDied","Data":"eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a"} Nov 01 01:18:45.711770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.711364 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5" event={"ID":"46be65a6-4f18-4e60-855c-6b08571e2981","Type":"ContainerDied","Data":"65f76930b2a7463ad50fef90dd69b039f6cc821756bfadb462f108992c55f563"} Nov 01 01:18:45.711770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.711377 4910 scope.go:122] "RemoveContainer" containerID="eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a" Nov 01 01:18:45.712855 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.712827 4910 generic.go:356] "Generic (PLEG): container finished" podID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerID="6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365" exitCode=0 Nov 01 01:18:45.712988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.712960 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" event={"ID":"a54f758e-a86e-44e8-aae6-eef45c5e6dda","Type":"ContainerDied","Data":"6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365"} Nov 01 01:18:45.713093 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.713007 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" event={"ID":"a54f758e-a86e-44e8-aae6-eef45c5e6dda","Type":"ContainerDied","Data":"60f7d4d76ef63befba9aaba7f2c05c1c797b976ca3461ef109c606ec1e63d051"} Nov 01 01:18:45.713093 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.713038 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" Nov 01 01:18:45.722102 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.722088 4910 scope.go:122] "RemoveContainer" containerID="eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a" Nov 01 01:18:45.722408 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:18:45.722380 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a\": container with ID starting with eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a not found: ID does not exist" containerID="eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a" Nov 01 01:18:45.722502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.722419 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a"} err="failed to get container status \"eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a\": rpc error: code = NotFound desc = could not find container \"eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a\": container with ID starting with eaf01e8f119296d15187a7398e60bab78801c378c724faea190f3f062033fc2a not found: ID does not exist" Nov 01 01:18:45.722502 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.722445 4910 scope.go:122] "RemoveContainer" containerID="6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365" Nov 01 01:18:45.730830 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.730811 4910 scope.go:122] "RemoveContainer" containerID="6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365" Nov 01 01:18:45.731073 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:18:45.731049 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365\": container with ID starting with 6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365 not found: ID does not exist" containerID="6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365" Nov 01 01:18:45.731155 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.731081 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365"} err="failed to get container status \"6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365\": rpc error: code = NotFound desc = could not find container \"6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365\": container with ID starting with 6c29810896705c145626ab75b5f98275cf72f5141671c628e6910702b26fa365 not found: ID does not exist" Nov 01 01:18:45.860638 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.860595 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 01 01:18:45.876747 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.876720 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt"] Nov 01 01:18:45.926431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.926354 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/distinct-2-w4-43a0cf-kserve-58b6b64795-h7hxt"] Nov 01 01:18:45.990474 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.990446 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5"] Nov 01 01:18:45.994886 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:45.994863 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-43a0cf/premium-w4-43a0cf-kserve-74457c7b9c-rcqb5"] Nov 01 01:18:46.190163 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:46.190092 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" path="/var/lib/kubelet/pods/46be65a6-4f18-4e60-855c-6b08571e2981/volumes" Nov 01 01:18:46.190504 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:46.190490 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" path="/var/lib/kubelet/pods/a54f758e-a86e-44e8-aae6-eef45c5e6dda/volumes" Nov 01 01:18:48.718793 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:48.718747 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" podUID="1b98851f-c830-4d22-a98b-8e1de94bde25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:18:48.719228 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:48.718755 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" podUID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:18:58.718528 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:58.718486 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" podUID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:18:58.719040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:18:58.718486 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" podUID="1b98851f-c830-4d22-a98b-8e1de94bde25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 01:19:01.821065 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:01.821034 4910 generic.go:356] "Generic (PLEG): container finished" podID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" containerID="91df9370a96e053d7b0a5ee094790b292c5bc505bec97eae3a4f448804ebac04" exitCode=0 Nov 01 01:19:01.821503 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:01.821102 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" event={"ID":"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4","Type":"ContainerDied","Data":"91df9370a96e053d7b0a5ee094790b292c5bc505bec97eae3a4f448804ebac04"} Nov 01 01:19:01.926206 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:01.926141 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 01:19:01.978860 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:01.978830 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-plugins-config-volume\") pod \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " Nov 01 01:19:01.979040 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:01.978876 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-kube-api-access-wqllt\" (UniqueName: \"kubernetes.io/projected/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-kube-api-access-wqllt\") pod \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\" (UID: \"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4\") " Nov 01 01:19:01.979236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:01.979213 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-plugins-config-volume" pod "4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" (UID: "4280bb5b-9384-4b89-93b1-f8e5bb62ceb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:19:01.981363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:01.981333 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-kube-api-access-wqllt" pod "4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" (UID: "4280bb5b-9384-4b89-93b1-f8e5bb62ceb4"). InnerVolumeSpecName "kube-api-access-wqllt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:19:02.027136 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.027112 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 01:19:02.079709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.079676 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b98851f-c830-4d22-a98b-8e1de94bde25-kube-api-access-h88ml\" (UniqueName: \"kubernetes.io/projected/1b98851f-c830-4d22-a98b-8e1de94bde25-kube-api-access-h88ml\") pod \"1b98851f-c830-4d22-a98b-8e1de94bde25\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " Nov 01 01:19:02.079901 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.079734 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b98851f-c830-4d22-a98b-8e1de94bde25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98851f-c830-4d22-a98b-8e1de94bde25-plugins-config-volume\") pod \"1b98851f-c830-4d22-a98b-8e1de94bde25\" (UID: \"1b98851f-c830-4d22-a98b-8e1de94bde25\") " Nov 01 01:19:02.079979 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.079947 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-plugins-config-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:02.079979 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.079959 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqllt\" (UniqueName: \"kubernetes.io/projected/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4-kube-api-access-wqllt\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:02.080143 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.080120 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b98851f-c830-4d22-a98b-8e1de94bde25-plugins-config-volume" pod "1b98851f-c830-4d22-a98b-8e1de94bde25" (UID: "1b98851f-c830-4d22-a98b-8e1de94bde25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:19:02.081988 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.081968 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b98851f-c830-4d22-a98b-8e1de94bde25-kube-api-access-h88ml" pod "1b98851f-c830-4d22-a98b-8e1de94bde25" (UID: "1b98851f-c830-4d22-a98b-8e1de94bde25"). InnerVolumeSpecName "kube-api-access-h88ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:19:02.180529 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.180446 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88ml\" (UniqueName: \"kubernetes.io/projected/1b98851f-c830-4d22-a98b-8e1de94bde25-kube-api-access-h88ml\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:02.180529 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.180485 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98851f-c830-4d22-a98b-8e1de94bde25-plugins-config-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:02.827770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.827734 4910 generic.go:356] "Generic (PLEG): container finished" podID="1b98851f-c830-4d22-a98b-8e1de94bde25" containerID="6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b" exitCode=0 Nov 01 01:19:02.828252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.827783 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" Nov 01 01:19:02.828252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.827810 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" event={"ID":"1b98851f-c830-4d22-a98b-8e1de94bde25","Type":"ContainerDied","Data":"6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b"} Nov 01 01:19:02.828252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.827867 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44" event={"ID":"1b98851f-c830-4d22-a98b-8e1de94bde25","Type":"ContainerDied","Data":"51ce024bfcda0facc8b1a74ef0483724d0b8d07596827317fa847b4bb4e46f42"} Nov 01 01:19:02.828252 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.827889 4910 scope.go:122] "RemoveContainer" containerID="6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b" Nov 01 01:19:02.838603 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.838572 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" event={"ID":"4280bb5b-9384-4b89-93b1-f8e5bb62ceb4","Type":"ContainerDied","Data":"2b7accce5d36318dd4a08e694f567b448fa5aa1249aee3dd073ca81a13da2fb5"} Nov 01 01:19:02.838785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.838665 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d" Nov 01 01:19:02.843754 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.843735 4910 scope.go:122] "RemoveContainer" containerID="6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b" Nov 01 01:19:02.844063 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:19:02.844045 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b\": container with ID starting with 6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b not found: ID does not exist" containerID="6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b" Nov 01 01:19:02.844128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.844071 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b"} err="failed to get container status \"6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b\": rpc error: code = NotFound desc = could not find container \"6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b\": container with ID starting with 6df77c298e02097aba33a8360e40b89f0a04699964230df882babeb6d3698a1b not found: ID does not exist" Nov 01 01:19:02.844128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.844090 4910 scope.go:122] "RemoveContainer" containerID="91df9370a96e053d7b0a5ee094790b292c5bc505bec97eae3a4f448804ebac04" Nov 01 01:19:02.994140 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:02.994062 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44"] Nov 01 01:19:03.044770 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:03.044744 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-43a0cf-d7bffc7dc-qxw44"] Nov 01 01:19:03.108451 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:03.108423 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d"] Nov 01 01:19:03.111519 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:03.111492 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-43a0cf-cfc96f556-bmc2d"] Nov 01 01:19:04.191379 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:04.191340 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b98851f-c830-4d22-a98b-8e1de94bde25" path="/var/lib/kubelet/pods/1b98851f-c830-4d22-a98b-8e1de94bde25/volumes" Nov 01 01:19:04.191792 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:04.191715 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" path="/var/lib/kubelet/pods/4280bb5b-9384-4b89-93b1-f8e5bb62ceb4/volumes" Nov 01 01:19:06.605239 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.605210 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-q25wr"] Nov 01 01:19:06.605795 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.605509 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerName="manager" containerID="cri-o://dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91" gracePeriod=30 Nov 01 01:19:06.861816 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.861754 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 01:19:06.865293 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.865269 4910 generic.go:356] "Generic (PLEG): container finished" podID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerID="dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91" exitCode=0 Nov 01 01:19:06.865417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.865310 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" Nov 01 01:19:06.865417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.865346 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" event={"ID":"b4a8bae4-5f29-4776-a979-eb6d0532e820","Type":"ContainerDied","Data":"dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91"} Nov 01 01:19:06.865417 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.865395 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb6cd4f9b-q25wr" event={"ID":"b4a8bae4-5f29-4776-a979-eb6d0532e820","Type":"ContainerDied","Data":"87eba920101707c482c09b4bdd5f3141642901785ebe9152fe14cbac326f4f58"} Nov 01 01:19:06.865538 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.865418 4910 scope.go:122] "RemoveContainer" containerID="dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91" Nov 01 01:19:06.876212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.876187 4910 scope.go:122] "RemoveContainer" containerID="7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637" Nov 01 01:19:06.884537 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.884518 4910 scope.go:122] "RemoveContainer" containerID="dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91" Nov 01 01:19:06.884821 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:19:06.884800 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91\": container with ID starting with dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91 not found: ID does not exist" containerID="dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91" Nov 01 01:19:06.884893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.884831 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91"} err="failed to get container status \"dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91\": rpc error: code = NotFound desc = could not find container \"dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91\": container with ID starting with dba068a3248328eeddf15cac5ab99323f81328739be6af72824ce00a90feaa91 not found: ID does not exist" Nov 01 01:19:06.884893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.884847 4910 scope.go:122] "RemoveContainer" containerID="7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637" Nov 01 01:19:06.885107 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:19:06.885080 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637\": container with ID starting with 7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637 not found: ID does not exist" containerID="7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637" Nov 01 01:19:06.885154 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.885105 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637"} err="failed to get container status \"7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637\": rpc error: code = NotFound desc = could not find container \"7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637\": container with ID starting with 7629018233ae0c8959ca416536be73f14029f67d58bacb5b28be5c17ff484637 not found: ID does not exist" Nov 01 01:19:06.911782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.911759 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a8bae4-5f29-4776-a979-eb6d0532e820-kube-api-access-pktq2\" (UniqueName: \"kubernetes.io/projected/b4a8bae4-5f29-4776-a979-eb6d0532e820-kube-api-access-pktq2\") pod \"b4a8bae4-5f29-4776-a979-eb6d0532e820\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " Nov 01 01:19:06.911882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.911837 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-metrics-certs\") pod \"b4a8bae4-5f29-4776-a979-eb6d0532e820\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " Nov 01 01:19:06.911882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.911857 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-webhook-cert\") pod \"b4a8bae4-5f29-4776-a979-eb6d0532e820\" (UID: \"b4a8bae4-5f29-4776-a979-eb6d0532e820\") " Nov 01 01:19:06.913947 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.913922 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a8bae4-5f29-4776-a979-eb6d0532e820-kube-api-access-pktq2" pod "b4a8bae4-5f29-4776-a979-eb6d0532e820" (UID: "b4a8bae4-5f29-4776-a979-eb6d0532e820"). InnerVolumeSpecName "kube-api-access-pktq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:19:06.914028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.913966 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-metrics-certs" pod "b4a8bae4-5f29-4776-a979-eb6d0532e820" (UID: "b4a8bae4-5f29-4776-a979-eb6d0532e820"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:19:06.914028 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:06.913993 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-webhook-cert" pod "b4a8bae4-5f29-4776-a979-eb6d0532e820" (UID: "b4a8bae4-5f29-4776-a979-eb6d0532e820"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:19:07.012831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:07.012801 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktq2\" (UniqueName: \"kubernetes.io/projected/b4a8bae4-5f29-4776-a979-eb6d0532e820-kube-api-access-pktq2\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:07.012831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:07.012826 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-metrics-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:07.012831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:07.012836 4910 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4a8bae4-5f29-4776-a979-eb6d0532e820-webhook-cert\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:19:07.286613 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:07.286578 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-q25wr"] Nov 01 01:19:07.290359 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:07.290336 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb6cd4f9b-q25wr"] Nov 01 01:19:08.190561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:08.190518 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" path="/var/lib/kubelet/pods/b4a8bae4-5f29-4776-a979-eb6d0532e820/volumes" Nov 01 01:19:13.102096 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:19:13.102063 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:19:33.448607 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:33.448576 4910 scope.go:122] "RemoveContainer" containerID="7d3f7afa1d87a98d225860053131799dcd6758f8d740190669544ce5c6f974a9" Nov 01 01:19:33.461557 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:19:33.461530 4910 scope.go:122] "RemoveContainer" containerID="e36afdb51ff92514825d1f368f61af9ba8df9bbb52803f41cf5ec1f317aed4d3" Nov 01 01:19:43.101993 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:19:43.101912 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:20:13.103227 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:20:13.103192 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:20:43.102664 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:20:43.102600 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:20:54.404343 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404312 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc"] Nov 01 01:20:54.404953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404891 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" containerName="payload-processing" Nov 01 01:20:54.404953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404911 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" containerName="payload-processing" Nov 01 01:20:54.404953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404927 4910 cpu_manager.go:414] "containerMap: removing container" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" Nov 01 01:20:54.404953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404934 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404960 4910 cpu_manager.go:414] "containerMap: removing container" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404967 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404980 4910 cpu_manager.go:414] "containerMap: removing container" podUID="71c8a8e2-fc7c-4309-b54a-03f297b6326d" containerName="revoke-keys" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404987 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c8a8e2-fc7c-4309-b54a-03f297b6326d" containerName="revoke-keys" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.404998 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerName="manager" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405005 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerName="manager" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405028 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerName="manager" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405035 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerName="manager" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405045 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1b98851f-c830-4d22-a98b-8e1de94bde25" containerName="payload-pre-processing" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405053 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b98851f-c830-4d22-a98b-8e1de94bde25" containerName="payload-pre-processing" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405066 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405073 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405170 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="46be65a6-4f18-4e60-855c-6b08571e2981" containerName="main" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405185 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c8a8e2-fc7c-4309-b54a-03f297b6326d" containerName="revoke-keys" Nov 01 01:20:54.405195 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405196 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerName="manager" Nov 01 01:20:54.405967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405206 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a8bae4-5f29-4776-a979-eb6d0532e820" containerName="manager" Nov 01 01:20:54.405967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405215 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b98851f-c830-4d22-a98b-8e1de94bde25" containerName="payload-pre-processing" Nov 01 01:20:54.405967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405223 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54f758e-a86e-44e8-aae6-eef45c5e6dda" containerName="main" Nov 01 01:20:54.405967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405233 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4280bb5b-9384-4b89-93b1-f8e5bb62ceb4" containerName="payload-processing" Nov 01 01:20:54.405967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.405242 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee64569-1fa7-4c78-b25f-32cefd90e832" containerName="istio-proxy" Nov 01 01:20:54.408781 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.408760 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545208 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545176 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545208 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545209 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c50f00-4c24-413d-9de5-ffab7aa1a279-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545414 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545335 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545414 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545369 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545427 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545448 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545467 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545489 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545498 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5dj\" (UniqueName: \"kubernetes.io/projected/94c50f00-4c24-413d-9de5-ffab7aa1a279-kube-api-access-9z5dj\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.545696 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.545611 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.612293 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.612263 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc"] Nov 01 01:20:54.646381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646352 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646385 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646405 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646425 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646444 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646487 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5dj\" (UniqueName: \"kubernetes.io/projected/94c50f00-4c24-413d-9de5-ffab7aa1a279-kube-api-access-9z5dj\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646546 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646533 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646916 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646572 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646916 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646596 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c50f00-4c24-413d-9de5-ffab7aa1a279-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.646916 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646842 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.647092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646919 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.647092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.646966 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.647092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.647027 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.647270 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.647252 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c50f00-4c24-413d-9de5-ffab7aa1a279-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.648991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.648971 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.649204 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.649185 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.707820 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.707754 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c50f00-4c24-413d-9de5-ffab7aa1a279-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.707967 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.707944 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5dj\" (UniqueName: \"kubernetes.io/projected/94c50f00-4c24-413d-9de5-ffab7aa1a279-kube-api-access-9z5dj\") pod \"maas-default-gateway-openshift-default-67bf986f99-8fzwc\" (UID: \"94c50f00-4c24-413d-9de5-ffab7aa1a279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:54.721587 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:54.721565 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:55.151282 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.151255 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc"] Nov 01 01:20:55.153000 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:20:55.152973 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c50f00_4c24_413d_9de5_ffab7aa1a279.slice/crio-27ff02b9e3379e8e12214e8cda6520127c3199d124e99f25e999521af32f3566 WatchSource:0}: Error finding container 27ff02b9e3379e8e12214e8cda6520127c3199d124e99f25e999521af32f3566: Status 404 returned error can't find the container with id 27ff02b9e3379e8e12214e8cda6520127c3199d124e99f25e999521af32f3566 Nov 01 01:20:55.155375 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.155344 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:20:55.155471 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.155411 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:20:55.155471 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.155442 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:20:55.469767 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.469669 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" event={"ID":"94c50f00-4c24-413d-9de5-ffab7aa1a279","Type":"ContainerStarted","Data":"eb681d883b1fa45dd376084ae1ad77828c229b91499054793811fc12a6cdc28d"} Nov 01 01:20:55.469767 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.469730 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" event={"ID":"94c50f00-4c24-413d-9de5-ffab7aa1a279","Type":"ContainerStarted","Data":"27ff02b9e3379e8e12214e8cda6520127c3199d124e99f25e999521af32f3566"} Nov 01 01:20:55.592270 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.592201 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podStartSLOduration=1.592182224 podStartE2EDuration="1.592182224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:20:55.590363649 +0000 UTC m=+3445.569945998" watchObservedRunningTime="2026-11-01 01:20:55.592182224 +0000 UTC m=+3445.571764560" Nov 01 01:20:55.722004 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:55.721908 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:20:56.723795 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:56.723764 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 01 01:20:56.724207 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:56.723815 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 01 01:20:57.722500 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:57.722462 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 01 01:20:57.722706 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:57.722518 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 01 01:20:58.722314 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:58.722280 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 01 01:20:58.722758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:58.722332 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 01 01:20:59.722428 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:59.722387 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 01 01:20:59.722893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:20:59.722468 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 01 01:21:00.722447 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:00.722386 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 01 01:21:00.723021 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:00.722458 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 01 01:21:01.722688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:01.722650 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 01 01:21:01.723097 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:01.722704 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 01 01:21:02.722323 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:02.722281 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 01 01:21:02.722543 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:02.722345 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 01 01:21:04.763834 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:04.763791 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 01:21:04.764376 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:04.763857 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 01:21:05.579031 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:05.578999 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:21:05.579407 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:05.579383 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:21:05.582790 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:05.582767 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" Nov 01 01:21:09.527007 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.526926 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw"] Nov 01 01:21:09.530472 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.530457 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611234 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611264 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611266 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabbbe39-9845-46b9-93d0-4e40d1001913-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611466 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611293 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jhd6\" (UniqueName: \"kubernetes.io/projected/fabbbe39-9845-46b9-93d0-4e40d1001913-kube-api-access-2jhd6\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611466 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611399 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611466 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611454 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611586 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611513 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611586 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611558 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611696 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611679 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.611745 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.611703 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712261 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712231 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712269 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712292 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712351 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712377 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712418 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712753 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712718 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712823 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712731 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712889 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712834 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.712889 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712840 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabbbe39-9845-46b9-93d0-4e40d1001913-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.713001 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712901 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jhd6\" (UniqueName: \"kubernetes.io/projected/fabbbe39-9845-46b9-93d0-4e40d1001913-kube-api-access-2jhd6\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.713001 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.712957 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.713272 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.713242 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.713393 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.713376 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fabbbe39-9845-46b9-93d0-4e40d1001913-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.715295 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.715275 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.715673 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.715600 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.743657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.743610 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw"] Nov 01 01:21:09.773888 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.773864 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fabbbe39-9845-46b9-93d0-4e40d1001913-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.774130 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.774110 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jhd6\" (UniqueName: \"kubernetes.io/projected/fabbbe39-9845-46b9-93d0-4e40d1001913-kube-api-access-2jhd6\") pod \"maas-default-gateway-openshift-default-67bf986f99-4pnsw\" (UID: \"fabbbe39-9845-46b9-93d0-4e40d1001913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:09.843843 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:09.843774 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:10.268905 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.268880 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw"] Nov 01 01:21:10.270363 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:21:10.270333 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabbbe39_9845_46b9_93d0_4e40d1001913.slice/crio-6f1c65e371104b0474b5ea025a8edc7ff649492832fc339dcf0c1b512d167a9e WatchSource:0}: Error finding container 6f1c65e371104b0474b5ea025a8edc7ff649492832fc339dcf0c1b512d167a9e: Status 404 returned error can't find the container with id 6f1c65e371104b0474b5ea025a8edc7ff649492832fc339dcf0c1b512d167a9e Nov 01 01:21:10.272710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.272675 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:21:10.272803 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.272741 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:21:10.272803 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.272773 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 01:21:10.570704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.569660 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" event={"ID":"fabbbe39-9845-46b9-93d0-4e40d1001913","Type":"ContainerStarted","Data":"aff3df1663f2de60af3617b4209c3e8665b2163c1af0ed5a835fcc1a2fb2cd0a"} Nov 01 01:21:10.570704 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.569738 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" event={"ID":"fabbbe39-9845-46b9-93d0-4e40d1001913","Type":"ContainerStarted","Data":"6f1c65e371104b0474b5ea025a8edc7ff649492832fc339dcf0c1b512d167a9e"} Nov 01 01:21:10.699744 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.699676 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" podStartSLOduration=1.6996563199999999 podStartE2EDuration="1.69965632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:21:10.689515526 +0000 UTC m=+3460.669097863" watchObservedRunningTime="2026-11-01 01:21:10.69965632 +0000 UTC m=+3460.679238655" Nov 01 01:21:10.844914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:10.844790 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:11.845346 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:11.845315 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4pnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 01 01:21:11.845764 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:11.845366 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" podUID="fabbbe39-9845-46b9-93d0-4e40d1001913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 01 01:21:12.844288 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:12.844255 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4pnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 01 01:21:12.844480 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:12.844305 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" podUID="fabbbe39-9845-46b9-93d0-4e40d1001913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 01 01:21:13.103021 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:21:13.102937 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:21:13.844667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:13.844562 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4pnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 01 01:21:13.844667 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:13.844615 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" podUID="fabbbe39-9845-46b9-93d0-4e40d1001913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 01 01:21:14.845034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:14.844993 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4pnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 01 01:21:14.845570 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:14.845057 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" podUID="fabbbe39-9845-46b9-93d0-4e40d1001913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 01 01:21:15.844199 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:15.844162 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4pnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 01 01:21:15.844407 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:15.844216 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" podUID="fabbbe39-9845-46b9-93d0-4e40d1001913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 01 01:21:16.844718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:16.844676 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4pnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 01 01:21:16.845184 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:16.844741 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" podUID="fabbbe39-9845-46b9-93d0-4e40d1001913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 01 01:21:18.351642 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:18.351586 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:18.352168 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:18.352051 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:18.352836 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:18.352817 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4pnsw" Nov 01 01:21:43.102548 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:21:43.102513 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:21:53.048674 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:53.048643 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq"] Nov 01 01:21:53.098588 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:53.098559 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1531a9a1-wd8hq"] Nov 01 01:21:54.190914 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:21:54.190883 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da41182-38b8-48d1-b83c-c81680dc95f5" path="/var/lib/kubelet/pods/2da41182-38b8-48d1-b83c-c81680dc95f5/volumes" Nov 01 01:22:07.048727 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:22:07.048694 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w"] Nov 01 01:22:07.072468 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:22:07.072442 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5350c629-nkz9w"] Nov 01 01:22:08.191212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:22:08.191167 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f079a9-3cf9-4ae2-be11-0754f442141b" path="/var/lib/kubelet/pods/f5f079a9-3cf9-4ae2-be11-0754f442141b/volumes" Nov 01 01:22:13.101787 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:22:13.101744 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:22:43.101554 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:22:43.101475 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:23:13.105081 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:23:13.105042 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:23:31.000396 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:31.000371 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:23:31.010298 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:31.010272 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:23:31.015671 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:31.015650 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:23:31.024839 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:31.024820 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:23:33.537940 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:33.537908 4910 scope.go:122] "RemoveContainer" containerID="694cd0755d0ca5d298af084e504a9d13b99d0c15b79450026eabbf5abbeecbc9" Nov 01 01:23:33.547657 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:33.547635 4910 scope.go:122] "RemoveContainer" containerID="4403db66eb58f5172e72cce9dba4d284d0197a338eff4d17be6d88c74213f331" Nov 01 01:23:33.555950 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:33.555932 4910 scope.go:122] "RemoveContainer" containerID="b9590152e7bd16e307d0afb7e1ed06ce04abdc80c8e862544f358c0850feff34" Nov 01 01:23:40.047082 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:40.047044 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m"] Nov 01 01:23:40.097290 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:40.097262 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ca337c-65c8m"] Nov 01 01:23:40.192527 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:23:40.192495 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c8a8e2-fc7c-4309-b54a-03f297b6326d" path="/var/lib/kubelet/pods/71c8a8e2-fc7c-4309-b54a-03f297b6326d/volumes" Nov 01 01:23:43.108677 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:23:43.108615 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:24:13.102244 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:24:13.102139 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:24:43.101979 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:24:43.101942 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:25:13.109501 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:25:13.109443 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:25:33.592403 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:25:33.592370 4910 scope.go:122] "RemoveContainer" containerID="46e5291ec211f629982512b65dfc4d965181417c9de235d5f2b9be2181a994dd" Nov 01 01:25:43.103239 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:25:43.103143 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:26:13.102601 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:26:13.102562 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:26:41.222597 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.222559 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs"] Nov 01 01:26:41.226348 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.226323 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.306694 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.306656 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs"] Nov 01 01:26:41.317096 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.317073 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6202e495-7854-419f-9c7f-d06bc26c8fe4-tls-certs\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.317215 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.317108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-dshm\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.317283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.317262 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-tmp-dir\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.317353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.317296 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpff\" (UniqueName: \"kubernetes.io/projected/6202e495-7854-419f-9c7f-d06bc26c8fe4-kube-api-access-4gpff\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.317353 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.317327 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-home\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.317473 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.317386 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-model-cache\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.331962 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.331940 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-43c48b\"/\"kube-root-ca.crt\"" Nov 01 01:26:41.331962 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.331943 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-43c48b\"/\"openshift-service-ca.crt\"" Nov 01 01:26:41.332155 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.332021 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-43c48b\"/\"default-dockercfg-x6c8g\"" Nov 01 01:26:41.332155 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.332039 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-43c48b\"/\"simulator-single-main-43c48b-kserve-self-signed-certs\"" Nov 01 01:26:41.418916 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.418879 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-dshm\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.418980 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-tmp-dir\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.419000 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpff\" (UniqueName: \"kubernetes.io/projected/6202e495-7854-419f-9c7f-d06bc26c8fe4-kube-api-access-4gpff\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.419015 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-home\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.419031 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-model-cache\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.419063 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6202e495-7854-419f-9c7f-d06bc26c8fe4-tls-certs\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.419389 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-home\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419477 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.419452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-tmp-dir\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.419526 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.419491 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-model-cache\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.421373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.421354 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-dshm\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.421640 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.421601 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6202e495-7854-419f-9c7f-d06bc26c8fe4-tls-certs\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.480511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.480445 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpff\" (UniqueName: \"kubernetes.io/projected/6202e495-7854-419f-9c7f-d06bc26c8fe4-kube-api-access-4gpff\") pod \"simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.538442 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.538415 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:41.934991 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:41.934958 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs"] Nov 01 01:26:41.935145 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:26:41.935047 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6202e495_7854_419f_9c7f_d06bc26c8fe4.slice/crio-b614aa74e42ab92a5bd391765c8df40544d9a024ac97a27eb99f6b7d3008ba29 WatchSource:0}: Error finding container b614aa74e42ab92a5bd391765c8df40544d9a024ac97a27eb99f6b7d3008ba29: Status 404 returned error can't find the container with id b614aa74e42ab92a5bd391765c8df40544d9a024ac97a27eb99f6b7d3008ba29 Nov 01 01:26:42.602272 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:42.602241 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" event={"ID":"6202e495-7854-419f-9c7f-d06bc26c8fe4","Type":"ContainerStarted","Data":"69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6"} Nov 01 01:26:42.602699 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:42.602289 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" event={"ID":"6202e495-7854-419f-9c7f-d06bc26c8fe4","Type":"ContainerStarted","Data":"b614aa74e42ab92a5bd391765c8df40544d9a024ac97a27eb99f6b7d3008ba29"} Nov 01 01:26:42.719605 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:42.719555 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" podStartSLOduration=1.719541481 podStartE2EDuration="1.719541481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:26:42.718514267 +0000 UTC m=+3792.698096605" watchObservedRunningTime="2026-11-01 01:26:42.719541481 +0000 UTC m=+3792.699123816" Nov 01 01:26:43.101961 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:26:43.101930 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:26:43.538731 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:43.538700 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:51.539324 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:51.539288 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:51.552596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:51.552572 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:26:51.664609 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:26:51.664582 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:27:13.103028 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:27:13.102968 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:27:41.651361 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:41.651327 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs"] Nov 01 01:27:43.102867 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:27:43.102837 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:27:56.672612 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:56.672558 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" podUID="6202e495-7854-419f-9c7f-d06bc26c8fe4" containerName="main" containerID="cri-o://69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6" gracePeriod=45 Nov 01 01:27:56.929878 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:56.929814 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:27:57.031951 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.031915 4910 generic.go:356] "Generic (PLEG): container finished" podID="6202e495-7854-419f-9c7f-d06bc26c8fe4" containerID="69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6" exitCode=0 Nov 01 01:27:57.032149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.031979 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" Nov 01 01:27:57.032149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.032004 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" event={"ID":"6202e495-7854-419f-9c7f-d06bc26c8fe4","Type":"ContainerDied","Data":"69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6"} Nov 01 01:27:57.032149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.032046 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs" event={"ID":"6202e495-7854-419f-9c7f-d06bc26c8fe4","Type":"ContainerDied","Data":"b614aa74e42ab92a5bd391765c8df40544d9a024ac97a27eb99f6b7d3008ba29"} Nov 01 01:27:57.032149 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.032063 4910 scope.go:122] "RemoveContainer" containerID="69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6" Nov 01 01:27:57.040997 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.040981 4910 scope.go:122] "RemoveContainer" containerID="69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6" Nov 01 01:27:57.041262 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:27:57.041242 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6\": container with ID starting with 69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6 not found: ID does not exist" containerID="69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6" Nov 01 01:27:57.041327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.041276 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6"} err="failed to get container status \"69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6\": rpc error: code = NotFound desc = could not find container \"69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6\": container with ID starting with 69798a83e2c645afd035031750f08be7ae3c667dc6b42aa5f52fdaadf7ffcff6 not found: ID does not exist" Nov 01 01:27:57.104605 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.104573 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6202e495-7854-419f-9c7f-d06bc26c8fe4-tls-certs\" (UniqueName: \"kubernetes.io/secret/6202e495-7854-419f-9c7f-d06bc26c8fe4-tls-certs\") pod \"6202e495-7854-419f-9c7f-d06bc26c8fe4\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " Nov 01 01:27:57.104605 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.104611 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-dshm\") pod \"6202e495-7854-419f-9c7f-d06bc26c8fe4\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " Nov 01 01:27:57.104865 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.104665 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-tmp-dir\") pod \"6202e495-7854-419f-9c7f-d06bc26c8fe4\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " Nov 01 01:27:57.104865 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.104721 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6202e495-7854-419f-9c7f-d06bc26c8fe4-kube-api-access-4gpff\" (UniqueName: \"kubernetes.io/projected/6202e495-7854-419f-9c7f-d06bc26c8fe4-kube-api-access-4gpff\") pod \"6202e495-7854-419f-9c7f-d06bc26c8fe4\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " Nov 01 01:27:57.104865 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.104812 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-model-cache\") pod \"6202e495-7854-419f-9c7f-d06bc26c8fe4\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " Nov 01 01:27:57.104865 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.104859 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-home\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-home\") pod \"6202e495-7854-419f-9c7f-d06bc26c8fe4\" (UID: \"6202e495-7854-419f-9c7f-d06bc26c8fe4\") " Nov 01 01:27:57.105092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.104985 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-tmp-dir" pod "6202e495-7854-419f-9c7f-d06bc26c8fe4" (UID: "6202e495-7854-419f-9c7f-d06bc26c8fe4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:27:57.105092 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.105058 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-model-cache" pod "6202e495-7854-419f-9c7f-d06bc26c8fe4" (UID: "6202e495-7854-419f-9c7f-d06bc26c8fe4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:27:57.105236 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.105215 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:57.105310 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.105248 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:57.105310 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.105212 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-home" pod "6202e495-7854-419f-9c7f-d06bc26c8fe4" (UID: "6202e495-7854-419f-9c7f-d06bc26c8fe4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:27:57.107038 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.107012 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6202e495-7854-419f-9c7f-d06bc26c8fe4-tls-certs" pod "6202e495-7854-419f-9c7f-d06bc26c8fe4" (UID: "6202e495-7854-419f-9c7f-d06bc26c8fe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:27:57.107038 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.107031 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-dshm" pod "6202e495-7854-419f-9c7f-d06bc26c8fe4" (UID: "6202e495-7854-419f-9c7f-d06bc26c8fe4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:27:57.107188 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.107170 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6202e495-7854-419f-9c7f-d06bc26c8fe4-kube-api-access-4gpff" pod "6202e495-7854-419f-9c7f-d06bc26c8fe4" (UID: "6202e495-7854-419f-9c7f-d06bc26c8fe4"). InnerVolumeSpecName "kube-api-access-4gpff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:27:57.206351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.206276 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:57.206351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.206303 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6202e495-7854-419f-9c7f-d06bc26c8fe4-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:57.206351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.206317 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6202e495-7854-419f-9c7f-d06bc26c8fe4-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:57.206351 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.206326 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpff\" (UniqueName: \"kubernetes.io/projected/6202e495-7854-419f-9c7f-d06bc26c8fe4-kube-api-access-4gpff\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:27:57.457588 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.457497 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs"] Nov 01 01:27:57.506219 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:57.506178 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-43c48b/simulator-single-main-43c48b-kserve-7d8d874c46-pzqcs"] Nov 01 01:27:58.190201 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:27:58.190158 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6202e495-7854-419f-9c7f-d06bc26c8fe4" path="/var/lib/kubelet/pods/6202e495-7854-419f-9c7f-d06bc26c8fe4/volumes" Nov 01 01:28:13.101956 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:28:13.101918 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:28:24.799230 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:24.799193 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q"] Nov 01 01:28:24.799774 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:24.799455 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" podUID="6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" containerName="manager" containerID="cri-o://0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14" gracePeriod=10 Nov 01 01:28:25.054758 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.054694 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 01:28:25.170118 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.170093 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-kube-api-access-mrt9d\" (UniqueName: \"kubernetes.io/projected/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-kube-api-access-mrt9d\") pod \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " Nov 01 01:28:25.170297 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.170215 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-extensions-socket-volume\") pod \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\" (UID: \"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744\") " Nov 01 01:28:25.170569 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.170547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-extensions-socket-volume" pod "6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" (UID: "6c3c9dd7-b9a9-4a37-b6b0-dd726d030744"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:28:25.172409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.172389 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-kube-api-access-mrt9d" pod "6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" (UID: "6c3c9dd7-b9a9-4a37-b6b0-dd726d030744"). InnerVolumeSpecName "kube-api-access-mrt9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:28:25.188114 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.188081 4910 generic.go:356] "Generic (PLEG): container finished" podID="6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" containerID="0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14" exitCode=0 Nov 01 01:28:25.188230 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.188195 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" Nov 01 01:28:25.188230 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.188162 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" event={"ID":"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744","Type":"ContainerDied","Data":"0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14"} Nov 01 01:28:25.188370 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.188248 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q" event={"ID":"6c3c9dd7-b9a9-4a37-b6b0-dd726d030744","Type":"ContainerDied","Data":"ca1fc937da0d1c56a40f83f4fcc4562dce7b4d6f0535155c764acdcd954a2799"} Nov 01 01:28:25.188370 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.188283 4910 scope.go:122] "RemoveContainer" containerID="0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14" Nov 01 01:28:25.204014 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.203987 4910 scope.go:122] "RemoveContainer" containerID="0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14" Nov 01 01:28:25.204339 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:28:25.204309 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14\": container with ID starting with 0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14 not found: ID does not exist" containerID="0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14" Nov 01 01:28:25.204441 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.204350 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14"} err="failed to get container status \"0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14\": rpc error: code = NotFound desc = could not find container \"0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14\": container with ID starting with 0198b5bddf62386de29417987323046833a02bccadb3a2af9c6a813abe0d0c14 not found: ID does not exist" Nov 01 01:28:25.271420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.271393 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrt9d\" (UniqueName: \"kubernetes.io/projected/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-kube-api-access-mrt9d\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:28:25.271420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.271418 4910 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744-extensions-socket-volume\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:28:25.356429 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.356345 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q"] Nov 01 01:28:25.361311 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:25.361287 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ncx6q"] Nov 01 01:28:26.190835 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:26.190800 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" path="/var/lib/kubelet/pods/6c3c9dd7-b9a9-4a37-b6b0-dd726d030744/volumes" Nov 01 01:28:31.039682 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:31.039652 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:28:31.049367 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:31.049339 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:28:31.054698 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:31.054681 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:28:31.064086 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:28:31.064069 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:28:43.101671 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:28:43.101576 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:29:13.102101 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:29:13.102061 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:29:28.491976 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.491935 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m"] Nov 01 01:29:28.494443 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.492357 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" containerName="manager" Nov 01 01:29:28.494443 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.492369 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" containerName="manager" Nov 01 01:29:28.494443 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.492392 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6202e495-7854-419f-9c7f-d06bc26c8fe4" containerName="main" Nov 01 01:29:28.494443 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.492397 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6202e495-7854-419f-9c7f-d06bc26c8fe4" containerName="main" Nov 01 01:29:28.494443 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.492465 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3c9dd7-b9a9-4a37-b6b0-dd726d030744" containerName="manager" Nov 01 01:29:28.494443 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.492479 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6202e495-7854-419f-9c7f-d06bc26c8fe4" containerName="main" Nov 01 01:29:28.495373 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.495350 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:28.545315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.545282 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m"] Nov 01 01:29:28.592759 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.592730 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-crws2\"" Nov 01 01:29:28.612212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.612178 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2c99\" (UniqueName: \"kubernetes.io/projected/f90b9ec6-0db6-4835-a570-87d82a2c23b2-kube-api-access-c2c99\") pod \"kuadrant-operator-controller-manager-678d87766f-nkb2m\" (UID: \"f90b9ec6-0db6-4835-a570-87d82a2c23b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:28.612332 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.612294 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90b9ec6-0db6-4835-a570-87d82a2c23b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nkb2m\" (UID: \"f90b9ec6-0db6-4835-a570-87d82a2c23b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:28.713410 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.713383 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90b9ec6-0db6-4835-a570-87d82a2c23b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nkb2m\" (UID: \"f90b9ec6-0db6-4835-a570-87d82a2c23b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:28.713683 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.713449 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2c99\" (UniqueName: \"kubernetes.io/projected/f90b9ec6-0db6-4835-a570-87d82a2c23b2-kube-api-access-c2c99\") pod \"kuadrant-operator-controller-manager-678d87766f-nkb2m\" (UID: \"f90b9ec6-0db6-4835-a570-87d82a2c23b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:28.713883 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.713863 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90b9ec6-0db6-4835-a570-87d82a2c23b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-nkb2m\" (UID: \"f90b9ec6-0db6-4835-a570-87d82a2c23b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:28.777659 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.777617 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2c99\" (UniqueName: \"kubernetes.io/projected/f90b9ec6-0db6-4835-a570-87d82a2c23b2-kube-api-access-c2c99\") pod \"kuadrant-operator-controller-manager-678d87766f-nkb2m\" (UID: \"f90b9ec6-0db6-4835-a570-87d82a2c23b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:28.809307 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:28.809283 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:29.286498 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:29.286471 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m"] Nov 01 01:29:29.287704 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:29:29.287680 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90b9ec6_0db6_4835_a570_87d82a2c23b2.slice/crio-9ec6e95e66996c15f9a046728234f3ad1714b8c9df68d3787290c158a4a5e006 WatchSource:0}: Error finding container 9ec6e95e66996c15f9a046728234f3ad1714b8c9df68d3787290c158a4a5e006: Status 404 returned error can't find the container with id 9ec6e95e66996c15f9a046728234f3ad1714b8c9df68d3787290c158a4a5e006 Nov 01 01:29:29.526037 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:29.526002 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" event={"ID":"f90b9ec6-0db6-4835-a570-87d82a2c23b2","Type":"ContainerStarted","Data":"0b39cf3ccc3c66f61c822ca696a3fc7d19277b1891e7d514ef340793c3d13e15"} Nov 01 01:29:29.526037 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:29.526058 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" event={"ID":"f90b9ec6-0db6-4835-a570-87d82a2c23b2","Type":"ContainerStarted","Data":"9ec6e95e66996c15f9a046728234f3ad1714b8c9df68d3787290c158a4a5e006"} Nov 01 01:29:29.526550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:29.526157 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:29.645868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:29.645776 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" podStartSLOduration=1.645763275 podStartE2EDuration="1.645763275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:29:29.644894608 +0000 UTC m=+3959.624476943" watchObservedRunningTime="2026-11-01 01:29:29.645763275 +0000 UTC m=+3959.625345609" Nov 01 01:29:40.533676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:29:40.533612 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-nkb2m" Nov 01 01:29:43.102791 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:29:43.102759 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:30:13.104044 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:30:13.104003 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:30:43.102020 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:30:43.101986 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:31:09.451737 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.451702 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt"] Nov 01 01:31:09.455517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.455496 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.532760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.532723 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-model-cache\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.532760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.532764 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8f70c9-5d48-49d8-973d-984260cb058a-tls-certs\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.532973 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.532852 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77sgr\" (UniqueName: \"kubernetes.io/projected/9e8f70c9-5d48-49d8-973d-984260cb058a-kube-api-access-77sgr\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.532973 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.532916 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-tmp-dir\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.533048 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.532991 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-home\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.533048 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.533033 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-dshm\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.557942 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.557915 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1048fc\"/\"kube-root-ca.crt\"" Nov 01 01:31:09.558081 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.557960 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1048fc\"/\"openshift-service-ca.crt\"" Nov 01 01:31:09.558081 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.558001 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1048fc\"/\"default-dockercfg-8mw45\"" Nov 01 01:31:09.560559 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.560541 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt"] Nov 01 01:31:09.560793 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.560779 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1048fc\"/\"simulator-single-main-1048fc-kserve-self-signed-certs\"" Nov 01 01:31:09.633671 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.633612 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77sgr\" (UniqueName: \"kubernetes.io/projected/9e8f70c9-5d48-49d8-973d-984260cb058a-kube-api-access-77sgr\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.633870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.633735 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-tmp-dir\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.633870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.633763 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-home\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.633870 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.633807 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-dshm\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.634029 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.633874 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-model-cache\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.634029 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.633901 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8f70c9-5d48-49d8-973d-984260cb058a-tls-certs\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.634246 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.634224 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-tmp-dir\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.634330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.634248 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-home\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.634330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.634299 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-model-cache\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.636244 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.636223 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-dshm\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.636409 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.636390 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8f70c9-5d48-49d8-973d-984260cb058a-tls-certs\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.696215 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.696181 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77sgr\" (UniqueName: \"kubernetes.io/projected/9e8f70c9-5d48-49d8-973d-984260cb058a-kube-api-access-77sgr\") pod \"simulator-single-main-1048fc-kserve-886974868-p6gjt\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:09.767369 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:09.767303 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:10.150499 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:10.150470 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt"] Nov 01 01:31:10.151608 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:31:10.151581 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e8f70c9_5d48_49d8_973d_984260cb058a.slice/crio-ee07742c86cd637b9952cdf77d4064c828c61170e3de6d2b89c61365dadf700b WatchSource:0}: Error finding container ee07742c86cd637b9952cdf77d4064c828c61170e3de6d2b89c61365dadf700b: Status 404 returned error can't find the container with id ee07742c86cd637b9952cdf77d4064c828c61170e3de6d2b89c61365dadf700b Nov 01 01:31:11.147537 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:11.147501 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" event={"ID":"9e8f70c9-5d48-49d8-973d-984260cb058a","Type":"ContainerStarted","Data":"062076d2a8fbced5c82910374166ad1d1362c14cd9d2f5b58dd75eab49c63f42"} Nov 01 01:31:11.147976 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:11.147552 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" event={"ID":"9e8f70c9-5d48-49d8-973d-984260cb058a","Type":"ContainerStarted","Data":"ee07742c86cd637b9952cdf77d4064c828c61170e3de6d2b89c61365dadf700b"} Nov 01 01:31:11.266105 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:11.266056 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" podStartSLOduration=2.266037289 podStartE2EDuration="2.266037289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:31:11.264059175 +0000 UTC m=+4061.243641521" watchObservedRunningTime="2026-11-01 01:31:11.266037289 +0000 UTC m=+4061.245619623" Nov 01 01:31:11.767563 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:11.767525 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:13.103565 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:31:13.103530 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:31:19.767687 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:19.767657 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:19.784134 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:19.784109 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:20.210600 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:31:20.210508 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:31:43.103757 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:31:43.103673 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:32:11.386474 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:11.386440 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt"] Nov 01 01:32:13.101546 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:32:13.101511 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:32:26.407259 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.407220 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" podUID="9e8f70c9-5d48-49d8-973d-984260cb058a" containerName="main" containerID="cri-o://062076d2a8fbced5c82910374166ad1d1362c14cd9d2f5b58dd75eab49c63f42" gracePeriod=45 Nov 01 01:32:26.570818 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.570787 4910 generic.go:356] "Generic (PLEG): container finished" podID="9e8f70c9-5d48-49d8-973d-984260cb058a" containerID="062076d2a8fbced5c82910374166ad1d1362c14cd9d2f5b58dd75eab49c63f42" exitCode=0 Nov 01 01:32:26.570998 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.570838 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" event={"ID":"9e8f70c9-5d48-49d8-973d-984260cb058a","Type":"ContainerDied","Data":"062076d2a8fbced5c82910374166ad1d1362c14cd9d2f5b58dd75eab49c63f42"} Nov 01 01:32:26.661709 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.661649 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:32:26.742174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742146 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-tmp-dir\") pod \"9e8f70c9-5d48-49d8-973d-984260cb058a\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " Nov 01 01:32:26.742314 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742180 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-home\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-home\") pod \"9e8f70c9-5d48-49d8-973d-984260cb058a\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " Nov 01 01:32:26.742314 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742244 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-dshm\") pod \"9e8f70c9-5d48-49d8-973d-984260cb058a\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " Nov 01 01:32:26.742314 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742285 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8f70c9-5d48-49d8-973d-984260cb058a-kube-api-access-77sgr\" (UniqueName: \"kubernetes.io/projected/9e8f70c9-5d48-49d8-973d-984260cb058a-kube-api-access-77sgr\") pod \"9e8f70c9-5d48-49d8-973d-984260cb058a\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " Nov 01 01:32:26.742506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742322 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-model-cache\") pod \"9e8f70c9-5d48-49d8-973d-984260cb058a\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " Nov 01 01:32:26.742506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742382 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8f70c9-5d48-49d8-973d-984260cb058a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8f70c9-5d48-49d8-973d-984260cb058a-tls-certs\") pod \"9e8f70c9-5d48-49d8-973d-984260cb058a\" (UID: \"9e8f70c9-5d48-49d8-973d-984260cb058a\") " Nov 01 01:32:26.742506 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742419 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-tmp-dir" pod "9e8f70c9-5d48-49d8-973d-984260cb058a" (UID: "9e8f70c9-5d48-49d8-973d-984260cb058a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:32:26.742695 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742527 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-home" pod "9e8f70c9-5d48-49d8-973d-984260cb058a" (UID: "9e8f70c9-5d48-49d8-973d-984260cb058a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:32:26.742760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742687 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-model-cache" pod "9e8f70c9-5d48-49d8-973d-984260cb058a" (UID: "9e8f70c9-5d48-49d8-973d-984260cb058a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:32:26.742823 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742762 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:26.742823 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.742782 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:26.744772 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.744749 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8f70c9-5d48-49d8-973d-984260cb058a-tls-certs" pod "9e8f70c9-5d48-49d8-973d-984260cb058a" (UID: "9e8f70c9-5d48-49d8-973d-984260cb058a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:32:26.745021 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.744997 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-dshm" pod "9e8f70c9-5d48-49d8-973d-984260cb058a" (UID: "9e8f70c9-5d48-49d8-973d-984260cb058a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:32:26.745116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.745023 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8f70c9-5d48-49d8-973d-984260cb058a-kube-api-access-77sgr" pod "9e8f70c9-5d48-49d8-973d-984260cb058a" (UID: "9e8f70c9-5d48-49d8-973d-984260cb058a"). InnerVolumeSpecName "kube-api-access-77sgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:32:26.844034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.844003 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:26.844034 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.844034 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77sgr\" (UniqueName: \"kubernetes.io/projected/9e8f70c9-5d48-49d8-973d-984260cb058a-kube-api-access-77sgr\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:26.844239 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.844046 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8f70c9-5d48-49d8-973d-984260cb058a-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:26.844239 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:26.844055 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8f70c9-5d48-49d8-973d-984260cb058a-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:32:27.578512 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:27.578483 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" Nov 01 01:32:27.579018 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:27.578526 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt" event={"ID":"9e8f70c9-5d48-49d8-973d-984260cb058a","Type":"ContainerDied","Data":"ee07742c86cd637b9952cdf77d4064c828c61170e3de6d2b89c61365dadf700b"} Nov 01 01:32:27.579018 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:27.578576 4910 scope.go:122] "RemoveContainer" containerID="062076d2a8fbced5c82910374166ad1d1362c14cd9d2f5b58dd75eab49c63f42" Nov 01 01:32:27.743644 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:27.743590 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt"] Nov 01 01:32:27.793615 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:27.793588 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1048fc/simulator-single-main-1048fc-kserve-886974868-p6gjt"] Nov 01 01:32:28.189983 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:32:28.189949 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8f70c9-5d48-49d8-973d-984260cb058a" path="/var/lib/kubelet/pods/9e8f70c9-5d48-49d8-973d-984260cb058a/volumes" Nov 01 01:32:43.103042 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:32:43.103003 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:33:13.103600 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:33:13.103566 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:33:22.391131 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.391100 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86"] Nov 01 01:33:22.391639 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.391491 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9e8f70c9-5d48-49d8-973d-984260cb058a" containerName="main" Nov 01 01:33:22.391639 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.391502 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8f70c9-5d48-49d8-973d-984260cb058a" containerName="main" Nov 01 01:33:22.391639 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.391569 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8f70c9-5d48-49d8-973d-984260cb058a" containerName="main" Nov 01 01:33:22.394707 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.394690 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.447432 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.447408 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 01:33:22.497147 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.497119 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c3b10ea0-kserve-self-signed-certs\"" Nov 01 01:33:22.497305 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.497180 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 01:33:22.497305 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.497214 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s6ps6\"" Nov 01 01:33:22.499939 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.499922 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86"] Nov 01 01:33:22.552771 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.552739 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-tmp-dir\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.552771 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.552776 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-model-cache\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.553041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.552802 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmp6t\" (UniqueName: \"kubernetes.io/projected/155f196e-bd0e-43fb-9962-658322749422-kube-api-access-hmp6t\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.553041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.553008 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-dshm\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.553136 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.553051 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f196e-bd0e-43fb-9962-658322749422-tls-certs\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.553177 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.553135 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-home\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654319 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654220 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-dshm\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654319 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654277 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f196e-bd0e-43fb-9962-658322749422-tls-certs\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654322 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-home\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654561 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654363 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-tmp-dir\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654785 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654763 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-model-cache\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654876 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654813 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmp6t\" (UniqueName: \"kubernetes.io/projected/155f196e-bd0e-43fb-9962-658322749422-kube-api-access-hmp6t\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654876 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654829 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-home\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.654876 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.654863 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-tmp-dir\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.655146 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.655127 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-model-cache\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.657097 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.657075 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-dshm\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.657179 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.657094 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f196e-bd0e-43fb-9962-658322749422-tls-certs\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:22.716178 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:22.716151 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmp6t\" (UniqueName: \"kubernetes.io/projected/155f196e-bd0e-43fb-9962-658322749422-kube-api-access-hmp6t\") pod \"e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:23.006504 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:23.006472 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:23.381669 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:23.381617 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86"] Nov 01 01:33:23.382737 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:33:23.382708 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod155f196e_bd0e_43fb_9962_658322749422.slice/crio-a91dc4accfba7e82078d249b7e6ed1149a1125fb0b427c59ad1846c2f627a229 WatchSource:0}: Error finding container a91dc4accfba7e82078d249b7e6ed1149a1125fb0b427c59ad1846c2f627a229: Status 404 returned error can't find the container with id a91dc4accfba7e82078d249b7e6ed1149a1125fb0b427c59ad1846c2f627a229 Nov 01 01:33:23.928466 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:23.928426 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" event={"ID":"155f196e-bd0e-43fb-9962-658322749422","Type":"ContainerStarted","Data":"3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d"} Nov 01 01:33:23.928902 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:23.928477 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" event={"ID":"155f196e-bd0e-43fb-9962-658322749422","Type":"ContainerStarted","Data":"a91dc4accfba7e82078d249b7e6ed1149a1125fb0b427c59ad1846c2f627a229"} Nov 01 01:33:24.007572 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:24.007543 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:24.046093 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:24.046046 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" podStartSLOduration=2.046033014 podStartE2EDuration="2.046033014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:33:24.044451582 +0000 UTC m=+4194.024033927" watchObservedRunningTime="2026-11-01 01:33:24.046033014 +0000 UTC m=+4194.025615348" Nov 01 01:33:31.080119 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:31.080092 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:33:31.090996 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:31.090975 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:33:31.096809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:31.096783 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:33:31.107135 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:31.107115 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:33:33.007418 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:33.007386 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:33.019807 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:33.019782 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:34.010420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:33:34.010392 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:33:43.105883 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:33:43.105852 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:34:13.106681 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:34:13.106617 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:34:29.176387 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:29.176351 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86"] Nov 01 01:34:43.102128 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:34:43.102040 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:34:44.196335 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.196300 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" podUID="155f196e-bd0e-43fb-9962-658322749422" containerName="main" containerID="cri-o://3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d" gracePeriod=45 Nov 01 01:34:44.441733 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.441703 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:34:44.499342 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499318 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-tmp-dir\") pod \"155f196e-bd0e-43fb-9962-658322749422\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " Nov 01 01:34:44.499517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499372 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155f196e-bd0e-43fb-9962-658322749422-kube-api-access-hmp6t\" (UniqueName: \"kubernetes.io/projected/155f196e-bd0e-43fb-9962-658322749422-kube-api-access-hmp6t\") pod \"155f196e-bd0e-43fb-9962-658322749422\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " Nov 01 01:34:44.499517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499432 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/155f196e-bd0e-43fb-9962-658322749422-tls-certs\" (UniqueName: \"kubernetes.io/secret/155f196e-bd0e-43fb-9962-658322749422-tls-certs\") pod \"155f196e-bd0e-43fb-9962-658322749422\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " Nov 01 01:34:44.499517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499451 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-dshm\") pod \"155f196e-bd0e-43fb-9962-658322749422\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " Nov 01 01:34:44.499517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499492 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-model-cache\") pod \"155f196e-bd0e-43fb-9962-658322749422\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " Nov 01 01:34:44.499746 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499572 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-home\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-home\") pod \"155f196e-bd0e-43fb-9962-658322749422\" (UID: \"155f196e-bd0e-43fb-9962-658322749422\") " Nov 01 01:34:44.499746 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499723 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-tmp-dir" pod "155f196e-bd0e-43fb-9962-658322749422" (UID: "155f196e-bd0e-43fb-9962-658322749422"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:34:44.499886 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.499866 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-model-cache" pod "155f196e-bd0e-43fb-9962-658322749422" (UID: "155f196e-bd0e-43fb-9962-658322749422"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:34:44.500100 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.500071 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-home" pod "155f196e-bd0e-43fb-9962-658322749422" (UID: "155f196e-bd0e-43fb-9962-658322749422"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:34:44.500244 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.500081 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-tmp-dir\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:34:44.500244 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.500160 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-model-cache\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:34:44.501848 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.501822 4910 generic.go:356] "Generic (PLEG): container finished" podID="155f196e-bd0e-43fb-9962-658322749422" containerID="3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d" exitCode=0 Nov 01 01:34:44.503684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.501878 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" Nov 01 01:34:44.503684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.501911 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" event={"ID":"155f196e-bd0e-43fb-9962-658322749422","Type":"ContainerDied","Data":"3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d"} Nov 01 01:34:44.503684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.501964 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86" event={"ID":"155f196e-bd0e-43fb-9962-658322749422","Type":"ContainerDied","Data":"a91dc4accfba7e82078d249b7e6ed1149a1125fb0b427c59ad1846c2f627a229"} Nov 01 01:34:44.503684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.501988 4910 scope.go:122] "RemoveContainer" containerID="3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d" Nov 01 01:34:44.503684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.502309 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-dshm" pod "155f196e-bd0e-43fb-9962-658322749422" (UID: "155f196e-bd0e-43fb-9962-658322749422"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 01:34:44.503684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.502451 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155f196e-bd0e-43fb-9962-658322749422-tls-certs" pod "155f196e-bd0e-43fb-9962-658322749422" (UID: "155f196e-bd0e-43fb-9962-658322749422"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:34:44.503684 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.502469 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155f196e-bd0e-43fb-9962-658322749422-kube-api-access-hmp6t" pod "155f196e-bd0e-43fb-9962-658322749422" (UID: "155f196e-bd0e-43fb-9962-658322749422"). InnerVolumeSpecName "kube-api-access-hmp6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:34:44.525203 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.525172 4910 scope.go:122] "RemoveContainer" containerID="3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d" Nov 01 01:34:44.525502 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:34:44.525481 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d\": container with ID starting with 3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d not found: ID does not exist" containerID="3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d" Nov 01 01:34:44.525602 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.525510 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d"} err="failed to get container status \"3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d\": rpc error: code = NotFound desc = could not find container \"3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d\": container with ID starting with 3d481eaba9ceab75e02df869db8455fdd0f2bd003dd8c6ccee6667e4aa280f9d not found: ID does not exist" Nov 01 01:34:44.601145 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.601108 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmp6t\" (UniqueName: \"kubernetes.io/projected/155f196e-bd0e-43fb-9962-658322749422-kube-api-access-hmp6t\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:34:44.601145 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.601144 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f196e-bd0e-43fb-9962-658322749422-tls-certs\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:34:44.601344 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.601158 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-dshm\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:34:44.601344 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.601171 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f196e-bd0e-43fb-9962-658322749422-home\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:34:44.921610 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.921576 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86"] Nov 01 01:34:44.927292 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:44.927261 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c3b10ea0-kserve-84d764467c-kfz86"] Nov 01 01:34:46.190548 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:34:46.190515 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155f196e-bd0e-43fb-9962-658322749422" path="/var/lib/kubelet/pods/155f196e-bd0e-43fb-9962-658322749422/volumes" Nov 01 01:35:13.102604 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:35:13.102559 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:35:43.102190 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:35:43.102153 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:36:07.773283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.773247 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z"] Nov 01 01:36:07.773841 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.773674 4910 cpu_manager.go:414] "containerMap: removing container" podUID="155f196e-bd0e-43fb-9962-658322749422" containerName="main" Nov 01 01:36:07.773841 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.773687 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155f196e-bd0e-43fb-9962-658322749422" containerName="main" Nov 01 01:36:07.773841 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.773760 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="155f196e-bd0e-43fb-9962-658322749422" containerName="main" Nov 01 01:36:07.776784 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.776768 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:07.831810 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.831784 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:36:07.831948 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.831857 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gr86n\"" Nov 01 01:36:07.884486 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.884454 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z"] Nov 01 01:36:07.901322 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.901290 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:07.901473 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.901335 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:07.901473 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:07.901437 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tmg\" (UniqueName: \"kubernetes.io/projected/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-kube-api-access-d6tmg\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.002619 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.002588 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tmg\" (UniqueName: \"kubernetes.io/projected/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-kube-api-access-d6tmg\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.002803 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.002700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.002803 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.002734 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.003479 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.003461 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.005355 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.005333 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.073662 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.073576 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tmg\" (UniqueName: \"kubernetes.io/projected/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-kube-api-access-d6tmg\") pod \"maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.089413 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.089391 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:08.461402 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:08.461375 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z"] Nov 01 01:36:08.462327 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:36:08.462302 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5c3d1d_0032_42d8_a01d_08b39ae5409b.slice/crio-2dd6104aaedd206e43185d5145a494aff8147a2e89cdf73a6418c73565bc8e6e WatchSource:0}: Error finding container 2dd6104aaedd206e43185d5145a494aff8147a2e89cdf73a6418c73565bc8e6e: Status 404 returned error can't find the container with id 2dd6104aaedd206e43185d5145a494aff8147a2e89cdf73a6418c73565bc8e6e Nov 01 01:36:09.050033 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:09.049917 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" event={"ID":"6b5c3d1d-0032-42d8-a01d-08b39ae5409b","Type":"ContainerStarted","Data":"457ad18d05db9487da564f58e4eb6ceb8b727626cd4849c9caf64e096d304524"} Nov 01 01:36:09.050033 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:09.049979 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" event={"ID":"6b5c3d1d-0032-42d8-a01d-08b39ae5409b","Type":"ContainerStarted","Data":"2dd6104aaedd206e43185d5145a494aff8147a2e89cdf73a6418c73565bc8e6e"} Nov 01 01:36:09.166701 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:09.166619 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" podStartSLOduration=2.166605319 podStartE2EDuration="2.166605319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:36:09.16511031 +0000 UTC m=+4359.144692671" watchObservedRunningTime="2026-11-01 01:36:09.166605319 +0000 UTC m=+4359.146187654" Nov 01 01:36:10.057397 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:10.057363 4910 generic.go:356] "Generic (PLEG): container finished" podID="6b5c3d1d-0032-42d8-a01d-08b39ae5409b" containerID="457ad18d05db9487da564f58e4eb6ceb8b727626cd4849c9caf64e096d304524" exitCode=0 Nov 01 01:36:10.057833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:10.057445 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" event={"ID":"6b5c3d1d-0032-42d8-a01d-08b39ae5409b","Type":"ContainerDied","Data":"457ad18d05db9487da564f58e4eb6ceb8b727626cd4849c9caf64e096d304524"} Nov 01 01:36:11.193749 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.193727 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:11.332315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.332241 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-openshift-ca\") pod \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " Nov 01 01:36:11.332315 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.332285 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-xks-ca\") pod \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " Nov 01 01:36:11.332511 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.332402 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-kube-api-access-d6tmg\" (UniqueName: \"kubernetes.io/projected/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-kube-api-access-d6tmg\") pod \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\" (UID: \"6b5c3d1d-0032-42d8-a01d-08b39ae5409b\") " Nov 01 01:36:11.332711 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.332681 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-openshift-ca" pod "6b5c3d1d-0032-42d8-a01d-08b39ae5409b" (UID: "6b5c3d1d-0032-42d8-a01d-08b39ae5409b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:36:11.334606 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.334581 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-xks-ca" pod "6b5c3d1d-0032-42d8-a01d-08b39ae5409b" (UID: "6b5c3d1d-0032-42d8-a01d-08b39ae5409b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:36:11.334742 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.334693 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-kube-api-access-d6tmg" pod "6b5c3d1d-0032-42d8-a01d-08b39ae5409b" (UID: "6b5c3d1d-0032-42d8-a01d-08b39ae5409b"). InnerVolumeSpecName "kube-api-access-d6tmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:36:11.433719 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.433695 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tmg\" (UniqueName: \"kubernetes.io/projected/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-kube-api-access-d6tmg\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:11.433719 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.433719 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:11.433893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:11.433728 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5c3d1d-0032-42d8-a01d-08b39ae5409b-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:36:12.068808 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:12.068781 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" Nov 01 01:36:12.068982 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:12.068816 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z" event={"ID":"6b5c3d1d-0032-42d8-a01d-08b39ae5409b","Type":"ContainerDied","Data":"2dd6104aaedd206e43185d5145a494aff8147a2e89cdf73a6418c73565bc8e6e"} Nov 01 01:36:12.068982 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:12.068864 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd6104aaedd206e43185d5145a494aff8147a2e89cdf73a6418c73565bc8e6e" Nov 01 01:36:13.102381 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:36:13.102345 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:36:26.428516 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:26.428483 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z"] Nov 01 01:36:26.478363 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:26.478334 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a684c9cb-c9v6z"] Nov 01 01:36:28.190089 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:36:28.190054 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5c3d1d-0032-42d8-a01d-08b39ae5409b" path="/var/lib/kubelet/pods/6b5c3d1d-0032-42d8-a01d-08b39ae5409b/volumes" Nov 01 01:36:43.106240 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:36:43.106192 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:37:13.102016 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:37:13.101981 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:37:43.103139 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:37:43.103053 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:38:13.102832 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:38:13.102789 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:38:21.197760 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.197728 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf"] Nov 01 01:38:21.198191 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.198116 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6b5c3d1d-0032-42d8-a01d-08b39ae5409b" containerName="revoke-keys" Nov 01 01:38:21.198191 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.198128 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5c3d1d-0032-42d8-a01d-08b39ae5409b" containerName="revoke-keys" Nov 01 01:38:21.198276 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.198216 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5c3d1d-0032-42d8-a01d-08b39ae5409b" containerName="revoke-keys" Nov 01 01:38:21.201259 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.201244 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.259908 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.259878 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf"] Nov 01 01:38:21.306180 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.306152 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:38:21.306328 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.306301 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gr86n\"" Nov 01 01:38:21.314445 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.314423 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.314549 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.314463 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.314602 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.314585 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvfn\" (UniqueName: \"kubernetes.io/projected/3cdb002c-b4b8-405c-b01e-19a47ae9918d-kube-api-access-8lvfn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.416017 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.415984 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.416210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.416032 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.416210 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.416091 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvfn\" (UniqueName: \"kubernetes.io/projected/3cdb002c-b4b8-405c-b01e-19a47ae9918d-kube-api-access-8lvfn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.416831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.416806 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.418696 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.418664 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.477466 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.477395 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvfn\" (UniqueName: \"kubernetes.io/projected/3cdb002c-b4b8-405c-b01e-19a47ae9918d-kube-api-access-8lvfn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.514385 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.514355 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:21.888504 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:21.888470 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf"] Nov 01 01:38:21.889332 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:38:21.889305 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdb002c_b4b8_405c_b01e_19a47ae9918d.slice/crio-524de79e2a825eb07e0f3da4ae4a9b407c19851458ce5d30e6c8a29dfe6626dc WatchSource:0}: Error finding container 524de79e2a825eb07e0f3da4ae4a9b407c19851458ce5d30e6c8a29dfe6626dc: Status 404 returned error can't find the container with id 524de79e2a825eb07e0f3da4ae4a9b407c19851458ce5d30e6c8a29dfe6626dc Nov 01 01:38:22.786748 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:22.786713 4910 generic.go:356] "Generic (PLEG): container finished" podID="3cdb002c-b4b8-405c-b01e-19a47ae9918d" containerID="228122a16fd8fc1db60676f9098eef3f8fce5e9e111ad6fde24993874b04f5da" exitCode=0 Nov 01 01:38:22.787116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:22.786800 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" event={"ID":"3cdb002c-b4b8-405c-b01e-19a47ae9918d","Type":"ContainerDied","Data":"228122a16fd8fc1db60676f9098eef3f8fce5e9e111ad6fde24993874b04f5da"} Nov 01 01:38:22.787116 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:22.786850 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" event={"ID":"3cdb002c-b4b8-405c-b01e-19a47ae9918d","Type":"ContainerStarted","Data":"524de79e2a825eb07e0f3da4ae4a9b407c19851458ce5d30e6c8a29dfe6626dc"} Nov 01 01:38:23.932976 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:23.932950 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:24.039868 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.039837 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cdb002c-b4b8-405c-b01e-19a47ae9918d-kube-api-access-8lvfn\" (UniqueName: \"kubernetes.io/projected/3cdb002c-b4b8-405c-b01e-19a47ae9918d-kube-api-access-8lvfn\") pod \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " Nov 01 01:38:24.040023 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.039914 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-xks-ca\") pod \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " Nov 01 01:38:24.040023 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.040013 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-openshift-ca\") pod \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\" (UID: \"3cdb002c-b4b8-405c-b01e-19a47ae9918d\") " Nov 01 01:38:24.040364 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.040329 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-openshift-ca" pod "3cdb002c-b4b8-405c-b01e-19a47ae9918d" (UID: "3cdb002c-b4b8-405c-b01e-19a47ae9918d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:38:24.042183 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.042157 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdb002c-b4b8-405c-b01e-19a47ae9918d-kube-api-access-8lvfn" pod "3cdb002c-b4b8-405c-b01e-19a47ae9918d" (UID: "3cdb002c-b4b8-405c-b01e-19a47ae9918d"). InnerVolumeSpecName "kube-api-access-8lvfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:38:24.042278 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.042196 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-xks-ca" pod "3cdb002c-b4b8-405c-b01e-19a47ae9918d" (UID: "3cdb002c-b4b8-405c-b01e-19a47ae9918d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:38:24.140893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.140860 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:24.140893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.140887 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvfn\" (UniqueName: \"kubernetes.io/projected/3cdb002c-b4b8-405c-b01e-19a47ae9918d-kube-api-access-8lvfn\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:24.140893 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.140899 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdb002c-b4b8-405c-b01e-19a47ae9918d-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:38:24.799294 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.799262 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" event={"ID":"3cdb002c-b4b8-405c-b01e-19a47ae9918d","Type":"ContainerDied","Data":"524de79e2a825eb07e0f3da4ae4a9b407c19851458ce5d30e6c8a29dfe6626dc"} Nov 01 01:38:24.799294 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.799295 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf" Nov 01 01:38:24.799517 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:24.799306 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="524de79e2a825eb07e0f3da4ae4a9b407c19851458ce5d30e6c8a29dfe6626dc" Nov 01 01:38:31.121637 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:31.121605 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:38:31.131912 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:31.131890 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:38:31.138999 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:31.138975 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76kvwg_b2c4449a-f5d7-4ea1-a3e3-d1cee8ceea0c/extract/0.log" Nov 01 01:38:31.147748 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:31.147731 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq7f9_252a2753-2639-4920-afaf-0fb0ca316d26/console-operator/2.log" Nov 01 01:38:43.102840 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:38:43.102804 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:38:45.391072 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:45.391040 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf"] Nov 01 01:38:45.440051 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:45.440022 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-930d1a-9rxrf"] Nov 01 01:38:46.191069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:38:46.191042 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cdb002c-b4b8-405c-b01e-19a47ae9918d" path="/var/lib/kubelet/pods/3cdb002c-b4b8-405c-b01e-19a47ae9918d/volumes" Nov 01 01:39:13.149696 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:39:13.149615 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:39:19.731965 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.731924 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq"] Nov 01 01:39:19.732579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.732537 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3cdb002c-b4b8-405c-b01e-19a47ae9918d" containerName="revoke-keys" Nov 01 01:39:19.732579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.732557 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cdb002c-b4b8-405c-b01e-19a47ae9918d" containerName="revoke-keys" Nov 01 01:39:19.732752 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.732732 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cdb002c-b4b8-405c-b01e-19a47ae9918d" containerName="revoke-keys" Nov 01 01:39:19.735833 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.735812 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.777692 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.777665 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.777813 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.777731 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kld\" (UniqueName: \"kubernetes.io/projected/87019b2c-0b6d-4495-83a5-b7318dc7f94c-kube-api-access-79kld\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.777877 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.777850 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.793676 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.793649 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq"] Nov 01 01:39:19.840530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.840501 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gr86n\"" Nov 01 01:39:19.840530 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.840522 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:39:19.878968 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.878943 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.879108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.878991 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kld\" (UniqueName: \"kubernetes.io/projected/87019b2c-0b6d-4495-83a5-b7318dc7f94c-kube-api-access-79kld\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.879108 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.879049 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.879685 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.879668 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.881440 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.881421 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:19.939845 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:19.939817 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kld\" (UniqueName: \"kubernetes.io/projected/87019b2c-0b6d-4495-83a5-b7318dc7f94c-kube-api-access-79kld\") pod \"maas-api-revoke-sub-f03629db-8rmkq\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:20.048134 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:20.048106 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:20.426325 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:20.426303 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq"] Nov 01 01:39:20.428365 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:39:20.428332 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87019b2c_0b6d_4495_83a5_b7318dc7f94c.slice/crio-3a7ca16315ce2c5b45a324a2a15cf5ebe96b8c2a209423fbba95099390dc715d WatchSource:0}: Error finding container 3a7ca16315ce2c5b45a324a2a15cf5ebe96b8c2a209423fbba95099390dc715d: Status 404 returned error can't find the container with id 3a7ca16315ce2c5b45a324a2a15cf5ebe96b8c2a209423fbba95099390dc715d Nov 01 01:39:21.168946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:21.168860 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" event={"ID":"87019b2c-0b6d-4495-83a5-b7318dc7f94c","Type":"ContainerStarted","Data":"dfa13a9ad1c03cd888da051f41a71f8afe4c40cdd861cf83f301ec88c1b3ac46"} Nov 01 01:39:21.168946 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:21.168916 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" event={"ID":"87019b2c-0b6d-4495-83a5-b7318dc7f94c","Type":"ContainerStarted","Data":"3a7ca16315ce2c5b45a324a2a15cf5ebe96b8c2a209423fbba95099390dc715d"} Nov 01 01:39:22.177267 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:22.177236 4910 generic.go:356] "Generic (PLEG): container finished" podID="87019b2c-0b6d-4495-83a5-b7318dc7f94c" containerID="dfa13a9ad1c03cd888da051f41a71f8afe4c40cdd861cf83f301ec88c1b3ac46" exitCode=0 Nov 01 01:39:22.177765 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:22.177329 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" event={"ID":"87019b2c-0b6d-4495-83a5-b7318dc7f94c","Type":"ContainerDied","Data":"dfa13a9ad1c03cd888da051f41a71f8afe4c40cdd861cf83f301ec88c1b3ac46"} Nov 01 01:39:23.333030 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.333003 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:23.410326 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.410295 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87019b2c-0b6d-4495-83a5-b7318dc7f94c-kube-api-access-79kld\" (UniqueName: \"kubernetes.io/projected/87019b2c-0b6d-4495-83a5-b7318dc7f94c-kube-api-access-79kld\") pod \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " Nov 01 01:39:23.410326 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.410326 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-openshift-ca\") pod \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " Nov 01 01:39:23.410570 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.410404 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-xks-ca\") pod \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\" (UID: \"87019b2c-0b6d-4495-83a5-b7318dc7f94c\") " Nov 01 01:39:23.410783 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.410760 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-openshift-ca" pod "87019b2c-0b6d-4495-83a5-b7318dc7f94c" (UID: "87019b2c-0b6d-4495-83a5-b7318dc7f94c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:39:23.412556 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.412527 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87019b2c-0b6d-4495-83a5-b7318dc7f94c-kube-api-access-79kld" pod "87019b2c-0b6d-4495-83a5-b7318dc7f94c" (UID: "87019b2c-0b6d-4495-83a5-b7318dc7f94c"). InnerVolumeSpecName "kube-api-access-79kld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:39:23.412663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.412588 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-xks-ca" pod "87019b2c-0b6d-4495-83a5-b7318dc7f94c" (UID: "87019b2c-0b6d-4495-83a5-b7318dc7f94c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:39:23.511825 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.511795 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:23.511825 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.511822 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kld\" (UniqueName: \"kubernetes.io/projected/87019b2c-0b6d-4495-83a5-b7318dc7f94c-kube-api-access-79kld\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:23.512008 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:23.511838 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87019b2c-0b6d-4495-83a5-b7318dc7f94c-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:24.189718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:24.189689 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" Nov 01 01:39:24.190651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:24.190598 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8rmkq" event={"ID":"87019b2c-0b6d-4495-83a5-b7318dc7f94c","Type":"ContainerDied","Data":"3a7ca16315ce2c5b45a324a2a15cf5ebe96b8c2a209423fbba95099390dc715d"} Nov 01 01:39:24.190651 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:24.190656 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a7ca16315ce2c5b45a324a2a15cf5ebe96b8c2a209423fbba95099390dc715d" Nov 01 01:39:40.360739 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.360704 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9"] Nov 01 01:39:40.363128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.361101 4910 cpu_manager.go:414] "containerMap: removing container" podUID="87019b2c-0b6d-4495-83a5-b7318dc7f94c" containerName="revoke-keys" Nov 01 01:39:40.363128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.361113 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87019b2c-0b6d-4495-83a5-b7318dc7f94c" containerName="revoke-keys" Nov 01 01:39:40.363128 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.361218 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="87019b2c-0b6d-4495-83a5-b7318dc7f94c" containerName="revoke-keys" Nov 01 01:39:40.364286 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.364267 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.469440 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.469407 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:39:40.469612 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.469470 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gr86n\"" Nov 01 01:39:40.473486 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.473460 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9"] Nov 01 01:39:40.498697 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.498669 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d20147-821e-414f-9408-980899d4ce03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.498851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.498718 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84cp\" (UniqueName: \"kubernetes.io/projected/25d20147-821e-414f-9408-980899d4ce03-kube-api-access-m84cp\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.498851 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.498798 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d20147-821e-414f-9408-980899d4ce03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.599643 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.599580 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d20147-821e-414f-9408-980899d4ce03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.599831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.599701 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84cp\" (UniqueName: \"kubernetes.io/projected/25d20147-821e-414f-9408-980899d4ce03-kube-api-access-m84cp\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.599831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.599754 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d20147-821e-414f-9408-980899d4ce03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.600437 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.600411 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d20147-821e-414f-9408-980899d4ce03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.602073 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.602051 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d20147-821e-414f-9408-980899d4ce03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.660699 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.660618 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84cp\" (UniqueName: \"kubernetes.io/projected/25d20147-821e-414f-9408-980899d4ce03-kube-api-access-m84cp\") pod \"maas-api-revoke-sub-0165d7d2-kvvp9\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:40.678064 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:40.678044 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:41.054718 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:41.054692 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9"] Nov 01 01:39:41.056400 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:39:41.056362 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d20147_821e_414f_9408_980899d4ce03.slice/crio-f81051e55a075a201541a156573b67bdaafbe4b40cebcef2bf100e35b032c617 WatchSource:0}: Error finding container f81051e55a075a201541a156573b67bdaafbe4b40cebcef2bf100e35b032c617: Status 404 returned error can't find the container with id f81051e55a075a201541a156573b67bdaafbe4b40cebcef2bf100e35b032c617 Nov 01 01:39:41.301179 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:41.301145 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" event={"ID":"25d20147-821e-414f-9408-980899d4ce03","Type":"ContainerStarted","Data":"f81051e55a075a201541a156573b67bdaafbe4b40cebcef2bf100e35b032c617"} Nov 01 01:39:42.307909 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:42.307880 4910 generic.go:356] "Generic (PLEG): container finished" podID="25d20147-821e-414f-9408-980899d4ce03" containerID="53ad9de887bda803a228339f3ce9d1b4b37c5ab8738c3a1cf7bcb1ba89913109" exitCode=0 Nov 01 01:39:42.308366 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:42.307972 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" event={"ID":"25d20147-821e-414f-9408-980899d4ce03","Type":"ContainerDied","Data":"53ad9de887bda803a228339f3ce9d1b4b37c5ab8738c3a1cf7bcb1ba89913109"} Nov 01 01:39:43.101939 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:39:43.101904 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:39:43.447797 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.447770 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:43.525029 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.524999 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d20147-821e-414f-9408-980899d4ce03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d20147-821e-414f-9408-980899d4ce03-maas-api-xks-ca\") pod \"25d20147-821e-414f-9408-980899d4ce03\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " Nov 01 01:39:43.525174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.525089 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d20147-821e-414f-9408-980899d4ce03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d20147-821e-414f-9408-980899d4ce03-maas-api-openshift-ca\") pod \"25d20147-821e-414f-9408-980899d4ce03\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " Nov 01 01:39:43.525174 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.525145 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d20147-821e-414f-9408-980899d4ce03-kube-api-access-m84cp\" (UniqueName: \"kubernetes.io/projected/25d20147-821e-414f-9408-980899d4ce03-kube-api-access-m84cp\") pod \"25d20147-821e-414f-9408-980899d4ce03\" (UID: \"25d20147-821e-414f-9408-980899d4ce03\") " Nov 01 01:39:43.525462 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.525434 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d20147-821e-414f-9408-980899d4ce03-maas-api-openshift-ca" pod "25d20147-821e-414f-9408-980899d4ce03" (UID: "25d20147-821e-414f-9408-980899d4ce03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:39:43.527291 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.527267 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d20147-821e-414f-9408-980899d4ce03-maas-api-xks-ca" pod "25d20147-821e-414f-9408-980899d4ce03" (UID: "25d20147-821e-414f-9408-980899d4ce03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:39:43.527385 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.527310 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d20147-821e-414f-9408-980899d4ce03-kube-api-access-m84cp" pod "25d20147-821e-414f-9408-980899d4ce03" (UID: "25d20147-821e-414f-9408-980899d4ce03"). InnerVolumeSpecName "kube-api-access-m84cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:39:43.626554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.626489 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d20147-821e-414f-9408-980899d4ce03-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:43.626554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.626515 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d20147-821e-414f-9408-980899d4ce03-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:43.626554 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:43.626526 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84cp\" (UniqueName: \"kubernetes.io/projected/25d20147-821e-414f-9408-980899d4ce03-kube-api-access-m84cp\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:44.320269 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:44.320234 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" Nov 01 01:39:44.320455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:44.320289 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kvvp9" event={"ID":"25d20147-821e-414f-9408-980899d4ce03","Type":"ContainerDied","Data":"f81051e55a075a201541a156573b67bdaafbe4b40cebcef2bf100e35b032c617"} Nov 01 01:39:44.320455 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:44.320343 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81051e55a075a201541a156573b67bdaafbe4b40cebcef2bf100e35b032c617" Nov 01 01:39:52.047782 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.047743 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64"] Nov 01 01:39:52.048273 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.048142 4910 cpu_manager.go:414] "containerMap: removing container" podUID="25d20147-821e-414f-9408-980899d4ce03" containerName="revoke-keys" Nov 01 01:39:52.048273 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.048153 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d20147-821e-414f-9408-980899d4ce03" containerName="revoke-keys" Nov 01 01:39:52.048273 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.048246 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d20147-821e-414f-9408-980899d4ce03" containerName="revoke-keys" Nov 01 01:39:52.051838 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.051817 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.110170 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.110137 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64"] Nov 01 01:39:52.156793 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.156756 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 01:39:52.157008 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.156940 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gr86n\"" Nov 01 01:39:52.203710 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.203677 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.203909 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.203742 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6tg\" (UniqueName: \"kubernetes.io/projected/0229c748-29d1-4282-8cb2-db6dedda9bcb-kube-api-access-vw6tg\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.203909 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.203892 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.305212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.305127 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.305212 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.305188 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.305408 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.305239 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6tg\" (UniqueName: \"kubernetes.io/projected/0229c748-29d1-4282-8cb2-db6dedda9bcb-kube-api-access-vw6tg\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.305779 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.305759 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.307808 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.307791 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.366769 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.366741 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6tg\" (UniqueName: \"kubernetes.io/projected/0229c748-29d1-4282-8cb2-db6dedda9bcb-kube-api-access-vw6tg\") pod \"maas-api-revoke-sub-6b19338f-bcg64\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:52.665016 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:52.664938 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:53.059272 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:53.059246 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64"] Nov 01 01:39:53.061026 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:39:53.061000 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0229c748_29d1_4282_8cb2_db6dedda9bcb.slice/crio-d8b046c47388cfe7570c9c0ce7ec6f3047481536120fdca27ee6330c2ddced66 WatchSource:0}: Error finding container d8b046c47388cfe7570c9c0ce7ec6f3047481536120fdca27ee6330c2ddced66: Status 404 returned error can't find the container with id d8b046c47388cfe7570c9c0ce7ec6f3047481536120fdca27ee6330c2ddced66 Nov 01 01:39:53.372143 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:53.372106 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" event={"ID":"0229c748-29d1-4282-8cb2-db6dedda9bcb","Type":"ContainerStarted","Data":"5cb9bb230a55c56e2a857788c904764934a413ea041528bbec85078e2e91fce9"} Nov 01 01:39:53.372283 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:53.372170 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" event={"ID":"0229c748-29d1-4282-8cb2-db6dedda9bcb","Type":"ContainerStarted","Data":"d8b046c47388cfe7570c9c0ce7ec6f3047481536120fdca27ee6330c2ddced66"} Nov 01 01:39:53.489568 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:53.489513 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" podStartSLOduration=1.489499513 podStartE2EDuration="1.489499513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:39:53.488118562 +0000 UTC m=+4583.467700894" watchObservedRunningTime="2026-11-01 01:39:53.489499513 +0000 UTC m=+4583.469081888" Nov 01 01:39:54.378937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:54.378903 4910 generic.go:356] "Generic (PLEG): container finished" podID="0229c748-29d1-4282-8cb2-db6dedda9bcb" containerID="5cb9bb230a55c56e2a857788c904764934a413ea041528bbec85078e2e91fce9" exitCode=0 Nov 01 01:39:54.379323 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:54.378965 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" event={"ID":"0229c748-29d1-4282-8cb2-db6dedda9bcb","Type":"ContainerDied","Data":"5cb9bb230a55c56e2a857788c904764934a413ea041528bbec85078e2e91fce9"} Nov 01 01:39:55.527324 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.527295 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:55.635300 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.635205 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-openshift-ca\") pod \"0229c748-29d1-4282-8cb2-db6dedda9bcb\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " Nov 01 01:39:55.635300 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.635272 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-xks-ca\") pod \"0229c748-29d1-4282-8cb2-db6dedda9bcb\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " Nov 01 01:39:55.635300 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.635293 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0229c748-29d1-4282-8cb2-db6dedda9bcb-kube-api-access-vw6tg\" (UniqueName: \"kubernetes.io/projected/0229c748-29d1-4282-8cb2-db6dedda9bcb-kube-api-access-vw6tg\") pod \"0229c748-29d1-4282-8cb2-db6dedda9bcb\" (UID: \"0229c748-29d1-4282-8cb2-db6dedda9bcb\") " Nov 01 01:39:55.635652 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.635600 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-openshift-ca" pod "0229c748-29d1-4282-8cb2-db6dedda9bcb" (UID: "0229c748-29d1-4282-8cb2-db6dedda9bcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 01:39:55.637513 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.637493 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-xks-ca" pod "0229c748-29d1-4282-8cb2-db6dedda9bcb" (UID: "0229c748-29d1-4282-8cb2-db6dedda9bcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 01:39:55.637579 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.637525 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0229c748-29d1-4282-8cb2-db6dedda9bcb-kube-api-access-vw6tg" pod "0229c748-29d1-4282-8cb2-db6dedda9bcb" (UID: "0229c748-29d1-4282-8cb2-db6dedda9bcb"). InnerVolumeSpecName "kube-api-access-vw6tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 01:39:55.735971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.735932 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-xks-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:55.735971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.735966 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6tg\" (UniqueName: \"kubernetes.io/projected/0229c748-29d1-4282-8cb2-db6dedda9bcb-kube-api-access-vw6tg\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:55.735971 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:55.735975 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229c748-29d1-4282-8cb2-db6dedda9bcb-maas-api-openshift-ca\") on node \"ip-10-0-137-150.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 01:39:56.391053 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:56.391014 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" event={"ID":"0229c748-29d1-4282-8cb2-db6dedda9bcb","Type":"ContainerDied","Data":"d8b046c47388cfe7570c9c0ce7ec6f3047481536120fdca27ee6330c2ddced66"} Nov 01 01:39:56.391053 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:56.391055 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bcg64" Nov 01 01:39:56.391331 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:39:56.391091 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b046c47388cfe7570c9c0ce7ec6f3047481536120fdca27ee6330c2ddced66" Nov 01 01:40:06.634848 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:06.634805 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8fzwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 01:40:06.635327 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:06.634883 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8fzwc" podUID="94c50f00-4c24-413d-9de5-ffab7aa1a279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 01:40:13.103306 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:40:13.103261 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:40:15.803665 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:15.803613 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sdwtw_a0c2335f-428b-4a64-9f23-bac748875f57/manager/0.log" Nov 01 01:40:16.423005 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:16.422976 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ng9v6_8469e4e1-96b0-4fce-a880-eaa35ff0157a/manager/0.log" Nov 01 01:40:17.823358 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:17.823331 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-w2ljf_e6308d92-d7fd-4085-a4dc-7a1971245c25/proxy/0.log" Nov 01 01:40:19.357831 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:19.357793 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kvvp9_25d20147-821e-414f-9408-980899d4ce03/revoke-keys/0.log" Nov 01 01:40:19.599124 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:19.599096 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-bcg64_0229c748-29d1-4282-8cb2-db6dedda9bcb/revoke-keys/0.log" Nov 01 01:40:19.843350 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:19.843320 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8rmkq_87019b2c-0b6d-4495-83a5-b7318dc7f94c/revoke-keys/0.log" Nov 01 01:40:20.076007 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:20.075975 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-6lcwq_7ad168f5-aacb-4793-87a6-088bd89161ec/postgres/0.log" Nov 01 01:40:22.223138 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:22.223104 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fp6nt_aa8d2336-53a8-4548-ab8a-05774f06ee89/manager/0.log" Nov 01 01:40:22.559056 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:22.559025 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tx8vx_4cda9f88-23b8-4d10-98b9-de79a789d559/manager/0.log" Nov 01 01:40:23.153663 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:23.153615 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nkb2m_f90b9ec6-0db6-4835-a570-87d82a2c23b2/manager/0.log" Nov 01 01:40:23.729361 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:23.729315 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vdplw_bbd2d693-1efb-4fd4-b23d-a9c4592ff185/manager/0.log" Nov 01 01:40:24.599610 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:24.599581 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd_4617ca5d-ea03-492c-a926-924ddf3b5860/istio-proxy/0.log" Nov 01 01:40:25.866397 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:25.866362 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-4pnsw_fabbbe39-9845-46b9-93d0-4e40d1001913/istio-proxy/0.log" Nov 01 01:40:26.660299 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:26.660264 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-8fzwc_94c50f00-4c24-413d-9de5-ffab7aa1a279/istio-proxy/0.log" Nov 01 01:40:27.150953 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:27.150923 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hg4hr_c16617ec-fda2-4f67-853b-fd58ff568d2c/istio-proxy/1.log" Nov 01 01:40:28.157193 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:28.157161 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw_0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7/istio-proxy/0.log" Nov 01 01:40:28.871743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:28.871706 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69cbfc858f-rq59s_0d85dc68-06e8-4036-93b1-ececa1a779c7/router/0.log" Nov 01 01:40:29.523863 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:29.523838 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hg4hr_c16617ec-fda2-4f67-853b-fd58ff568d2c/istio-proxy/0.log" Nov 01 01:40:42.957291 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:42.957217 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sdwtw_a0c2335f-428b-4a64-9f23-bac748875f57/manager/0.log" Nov 01 01:40:43.102290 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:40:43.102258 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:40:43.535969 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:43.535940 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ng9v6_8469e4e1-96b0-4fce-a880-eaa35ff0157a/manager/0.log" Nov 01 01:40:44.802126 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:44.802099 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-w2ljf_e6308d92-d7fd-4085-a4dc-7a1971245c25/proxy/0.log" Nov 01 01:40:45.947006 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:45.946978 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kvvp9_25d20147-821e-414f-9408-980899d4ce03/revoke-keys/0.log" Nov 01 01:40:46.179809 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:46.179779 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-bcg64_0229c748-29d1-4282-8cb2-db6dedda9bcb/revoke-keys/0.log" Nov 01 01:40:46.409917 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:46.409886 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8rmkq_87019b2c-0b6d-4495-83a5-b7318dc7f94c/revoke-keys/0.log" Nov 01 01:40:46.633438 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:46.633411 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-6lcwq_7ad168f5-aacb-4793-87a6-088bd89161ec/postgres/0.log" Nov 01 01:40:48.272196 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:48.272164 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fp6nt_aa8d2336-53a8-4548-ab8a-05774f06ee89/manager/0.log" Nov 01 01:40:48.598641 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:48.598540 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tx8vx_4cda9f88-23b8-4d10-98b9-de79a789d559/manager/0.log" Nov 01 01:40:49.064398 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:49.064368 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nkb2m_f90b9ec6-0db6-4835-a570-87d82a2c23b2/manager/0.log" Nov 01 01:40:49.699095 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:49.699068 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vdplw_bbd2d693-1efb-4fd4-b23d-a9c4592ff185/manager/0.log" Nov 01 01:40:50.294013 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:50.293982 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd_4617ca5d-ea03-492c-a926-924ddf3b5860/istio-proxy/0.log" Nov 01 01:40:51.608406 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:51.608378 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-4pnsw_fabbbe39-9845-46b9-93d0-4e40d1001913/istio-proxy/0.log" Nov 01 01:40:52.323206 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:52.323181 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-8fzwc_94c50f00-4c24-413d-9de5-ffab7aa1a279/istio-proxy/0.log" Nov 01 01:40:52.829302 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:52.829267 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hg4hr_c16617ec-fda2-4f67-853b-fd58ff568d2c/istio-proxy/1.log" Nov 01 01:40:53.845730 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:53.845695 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-vrmtw_0599ef9c-ba0c-4f5c-a5b6-8af53d4d1dd7/istio-proxy/0.log" Nov 01 01:40:54.587024 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:40:54.586992 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69cbfc858f-rq59s_0d85dc68-06e8-4036-93b1-ececa1a779c7/router/0.log" Nov 01 01:41:13.103415 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:41:13.103378 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:41:43.102150 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:41:43.102114 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:42:06.323861 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.323826 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd5lg/must-gather-z457n"] Nov 01 01:42:06.324381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.324242 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0229c748-29d1-4282-8cb2-db6dedda9bcb" containerName="revoke-keys" Nov 01 01:42:06.324381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.324252 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0229c748-29d1-4282-8cb2-db6dedda9bcb" containerName="revoke-keys" Nov 01 01:42:06.324381 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.324315 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0229c748-29d1-4282-8cb2-db6dedda9bcb" containerName="revoke-keys" Nov 01 01:42:06.327547 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.327531 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:06.347161 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.347134 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa19e0e8-ee98-4f93-8476-f43d64a89755-must-gather-output\") pod \"must-gather-z457n\" (UID: \"aa19e0e8-ee98-4f93-8476-f43d64a89755\") " pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:06.347336 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.347203 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8nzx\" (UniqueName: \"kubernetes.io/projected/aa19e0e8-ee98-4f93-8476-f43d64a89755-kube-api-access-g8nzx\") pod \"must-gather-z457n\" (UID: \"aa19e0e8-ee98-4f93-8476-f43d64a89755\") " pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:06.379937 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.379905 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd5lg\"/\"openshift-service-ca.crt\"" Nov 01 01:42:06.430121 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.430083 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd5lg\"/\"kube-root-ca.crt\"" Nov 01 01:42:06.430298 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.430167 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rd5lg\"/\"default-dockercfg-6z6n4\"" Nov 01 01:42:06.433020 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.432998 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/must-gather-z457n"] Nov 01 01:42:06.447913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.447878 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa19e0e8-ee98-4f93-8476-f43d64a89755-must-gather-output\") pod \"must-gather-z457n\" (UID: \"aa19e0e8-ee98-4f93-8476-f43d64a89755\") " pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:06.448071 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.447961 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nzx\" (UniqueName: \"kubernetes.io/projected/aa19e0e8-ee98-4f93-8476-f43d64a89755-kube-api-access-g8nzx\") pod \"must-gather-z457n\" (UID: \"aa19e0e8-ee98-4f93-8476-f43d64a89755\") " pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:06.448221 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.448204 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa19e0e8-ee98-4f93-8476-f43d64a89755-must-gather-output\") pod \"must-gather-z457n\" (UID: \"aa19e0e8-ee98-4f93-8476-f43d64a89755\") " pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:06.509466 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.509421 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8nzx\" (UniqueName: \"kubernetes.io/projected/aa19e0e8-ee98-4f93-8476-f43d64a89755-kube-api-access-g8nzx\") pod \"must-gather-z457n\" (UID: \"aa19e0e8-ee98-4f93-8476-f43d64a89755\") " pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:06.641550 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:06.641468 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/must-gather-z457n" Nov 01 01:42:07.013993 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:07.013970 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/must-gather-z457n"] Nov 01 01:42:07.015754 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:42:07.015728 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa19e0e8_ee98_4f93_8476_f43d64a89755.slice/crio-9bd4cbe28ff36075ee333bf3cca6b48bd91d59d5d17575cffc9adfd63887d07b WatchSource:0}: Error finding container 9bd4cbe28ff36075ee333bf3cca6b48bd91d59d5d17575cffc9adfd63887d07b: Status 404 returned error can't find the container with id 9bd4cbe28ff36075ee333bf3cca6b48bd91d59d5d17575cffc9adfd63887d07b Nov 01 01:42:07.017688 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:07.017669 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 01:42:07.119386 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:07.119350 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/must-gather-z457n" event={"ID":"aa19e0e8-ee98-4f93-8476-f43d64a89755","Type":"ContainerStarted","Data":"9bd4cbe28ff36075ee333bf3cca6b48bd91d59d5d17575cffc9adfd63887d07b"} Nov 01 01:42:09.136923 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:09.136839 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/must-gather-z457n" event={"ID":"aa19e0e8-ee98-4f93-8476-f43d64a89755","Type":"ContainerStarted","Data":"0f4d9847ea0b2c258b01083d95aa94951c7ea25ddcd08b5635c986a0b4964813"} Nov 01 01:42:09.136923 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:09.136881 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/must-gather-z457n" event={"ID":"aa19e0e8-ee98-4f93-8476-f43d64a89755","Type":"ContainerStarted","Data":"755fcf40b8a31de0d25b9f0c54f2c1b8318b5d6745d327188177f0a35f0afe53"} Nov 01 01:42:09.253330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:09.253276 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rd5lg/must-gather-z457n" podStartSLOduration=1.9136934060000002 podStartE2EDuration="3.253261783s" totalImagesPullingTime="1.339568377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 01:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:42:09.252045275 +0000 UTC m=+4719.231627610" watchObservedRunningTime="2026-11-01 01:42:09.253261783 +0000 UTC m=+4719.232844117" Nov 01 01:42:13.106090 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:42:13.106049 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:42:15.326741 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:15.326689 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m45bc_4454de50-086b-4842-8e00-d2ffae8344d2/global-pull-secret-syncer/0.log" Nov 01 01:42:17.164913 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:17.164882 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7btpl_c5b7952d-24a8-40c5-a755-3aeb8269714b/connection-checker/0.log" Nov 01 01:42:17.606258 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:17.605999 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fxsjd_5cfa57d0-764b-42fb-ad36-9ca6dc472766/connection-checker/0.log" Nov 01 01:42:18.039302 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:18.039273 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gsr76_b7e4c8da-dd72-438d-bd5b-a81728051a95/connection-checker/0.log" Nov 01 01:42:19.521661 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:19.519557 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vtb29_60389df0-3bb0-42fd-9a5b-40bfa22bacf5/konnectivity-agent/0.log" Nov 01 01:42:19.900138 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:19.900018 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-150.us-west-1.compute.internal_f0c4975681f29214750bf74479eb4135/haproxy/0.log" Nov 01 01:42:25.134845 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.134812 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fp6nt_aa8d2336-53a8-4548-ab8a-05774f06ee89/manager/0.log" Nov 01 01:42:25.393125 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.393035 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2"] Nov 01 01:42:25.400383 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.400359 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.457088 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.457046 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2"] Nov 01 01:42:25.519742 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.519707 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnn7r\" (UniqueName: \"kubernetes.io/projected/091a5dd9-753e-4710-979c-f5c63b04d081-kube-api-access-gnn7r\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.519742 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.519748 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-lib-modules\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.519994 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.519880 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-proc\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.519994 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.519932 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-podres\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.520069 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.519991 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-sys\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.556249 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.556221 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tx8vx_4cda9f88-23b8-4d10-98b9-de79a789d559/manager/0.log" Nov 01 01:42:25.620850 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.620822 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnn7r\" (UniqueName: \"kubernetes.io/projected/091a5dd9-753e-4710-979c-f5c63b04d081-kube-api-access-gnn7r\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.620857 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-lib-modules\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.620911 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-proc\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.620947 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-podres\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.620963 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-sys\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621041 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.621028 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-lib-modules\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621251 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.621049 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-sys\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621251 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.621028 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-proc\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.621251 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.621062 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/091a5dd9-753e-4710-979c-f5c63b04d081-podres\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.682420 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.682360 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnn7r\" (UniqueName: \"kubernetes.io/projected/091a5dd9-753e-4710-979c-f5c63b04d081-kube-api-access-gnn7r\") pod \"perf-node-gather-daemonset-ltrs2\" (UID: \"091a5dd9-753e-4710-979c-f5c63b04d081\") " pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:25.713425 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:25.713400 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:26.090835 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:26.090810 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2"] Nov 01 01:42:26.091982 ip-10-0-137-150 kubenswrapper[4910]: W1101 01:42:26.091948 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod091a5dd9_753e_4710_979c_f5c63b04d081.slice/crio-5beb19eb1f9414cc02d1d58168337d23c6fe18a114d6804a7ee780a26e0b87d4 WatchSource:0}: Error finding container 5beb19eb1f9414cc02d1d58168337d23c6fe18a114d6804a7ee780a26e0b87d4: Status 404 returned error can't find the container with id 5beb19eb1f9414cc02d1d58168337d23c6fe18a114d6804a7ee780a26e0b87d4 Nov 01 01:42:26.296415 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:26.296378 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" event={"ID":"091a5dd9-753e-4710-979c-f5c63b04d081","Type":"ContainerStarted","Data":"93a7af308bba205fb207b8946f6555dd70c34ebd10bc878d56a44e262169d01c"} Nov 01 01:42:26.296882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:26.296446 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" event={"ID":"091a5dd9-753e-4710-979c-f5c63b04d081","Type":"ContainerStarted","Data":"5beb19eb1f9414cc02d1d58168337d23c6fe18a114d6804a7ee780a26e0b87d4"} Nov 01 01:42:26.296882 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:26.296501 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:26.415568 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:26.415466 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" podStartSLOduration=1.415452294 podStartE2EDuration="1.415452294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 01:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 01:42:26.412346628 +0000 UTC m=+4736.391928963" watchObservedRunningTime="2026-11-01 01:42:26.415452294 +0000 UTC m=+4736.395034626" Nov 01 01:42:26.456885 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:26.456855 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-nkb2m_f90b9ec6-0db6-4835-a570-87d82a2c23b2/manager/0.log" Nov 01 01:42:27.383330 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:27.383299 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vdplw_bbd2d693-1efb-4fd4-b23d-a9c4592ff185/manager/0.log" Nov 01 01:42:32.314046 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:32.314015 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rd5lg/perf-node-gather-daemonset-ltrs2" Nov 01 01:42:32.911596 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:32.911562 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-2vfwz_1ba5db8b-e910-4ba0-b604-d1a8b393d1f7/cluster-monitoring-operator/0.log" Nov 01 01:42:33.773386 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:33.773340 4910 scope.go:122] "RemoveContainer" containerID="457ad18d05db9487da564f58e4eb6ceb8b727626cd4849c9caf64e096d304524" Nov 01 01:42:36.777431 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:36.777392 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4xhx_4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c/node-exporter/0.log" Nov 01 01:42:37.208575 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:37.208495 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4xhx_4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c/kube-rbac-proxy/0.log" Nov 01 01:42:37.627743 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:37.627717 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4xhx_4e0bb0df-b83c-4a00-975d-fdf9b2ffb50c/init-textfile/0.log" Nov 01 01:42:43.101970 ip-10-0-137-150 kubenswrapper[4910]: E1101 01:42:43.101923 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k2s97" containerName="extractor" Nov 01 01:42:48.442660 ip-10-0-137-150 kubenswrapper[4910]: I1101 01:42:48.441073 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccmbdjd_4617ca5d-ea03-492c-a926-924ddf3b5860/istio-proxy/0.log"